mirror of
https://github.com/foomo/foomo-docs.git
synced 2025-10-16 12:35:40 +00:00
Deploy website - based on d0dc058b03
This commit is contained in:
gh-actions
parent
590de121c6
commit
7d5e0b5c1a
10
404.html
10
404.html
@ -9,13 +9,13 @@
|
||||
|
||||
|
||||
<link rel="search" type="application/opensearchdescription+xml" title="foomo project docs" href="/opensearch.xml"><link rel="stylesheet" href="/assets/css/styles.45fe17b6.css">
|
||||
<link rel="preload" href="/assets/js/runtime~main.926d63f1.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.84c4b0c2.js" as="script">
|
||||
<link rel="preload" href="/assets/js/runtime~main.881e34e7.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.0e9b6dcd.js" as="script">
|
||||
</head>
|
||||
<body class="navigation-with-keyboard">
|
||||
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus">
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/project-management">PM</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a class="navbar__item navbar__link" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><main class="container margin-vert--xl"><div class="row"><div class="col col--6 col--offset-3"><h1 class="hero__title">Page Not Found</h1><p>We could not find what you were looking for.</p><p>Please contact the owner of the site that linked you to the original URL and let them know their link is broken.</p></div></div></main></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.926d63f1.js"></script>
|
||||
<script src="/assets/js/main.84c4b0c2.js"></script>
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a class="navbar__item navbar__link" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><main class="container margin-vert--xl"><div class="row"><div class="col col--6 col--offset-3"><h1 class="hero__title">Page Not Found</h1><p>We could not find what you were looking for.</p><p>Please contact the owner of the site that linked you to the original URL and let them know their link is broken.</p></div></div></main></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.881e34e7.js"></script>
|
||||
<script src="/assets/js/main.0e9b6dcd.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because one or more lines are too long
@ -1 +0,0 @@
|
||||
"use strict";(self.webpackChunkfoomo=self.webpackChunkfoomo||[]).push([[8124],{3905:(e,t,r)=>{r.d(t,{Zo:()=>s,kt:()=>d});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function a(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t<arguments.length;t++){var r=null!=arguments[t]?arguments[t]:{};t%2?a(Object(r),!0).forEach((function(t){o(e,t,r[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(r)):a(Object(r)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(r,t))}))}return e}function c(e,t){if(null==e)return{};var r,n,o=function(e,t){if(null==e)return{};var r,n,o={},a=Object.keys(e);for(n=0;n<a.length;n++)r=a[n],t.indexOf(r)>=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(n=0;n<a.length;n++)r=a[n],t.indexOf(r)>=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var p=n.createContext({}),l=function(e){var t=n.useContext(p),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},s=function(e){var t=l(e.components);return n.createElement(p.Provider,{value:t},e.children)},m="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},f=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,a=e.originalType,p=e.parentName,s=c(e,["components","mdxType","originalType","parentName"]),m=l(r),f=o,d=m["".concat(p,".").concat(f)]||m[f]||u[f]||a;return r?n.createElement(d,i(i({ref:t},s),{},{components:r})):n.createElement(d,i({ref:t},s))}));function d(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=r.length,i=new Array(a);i[0]=f;var c={};for(var p in t)hasOwnProperty.call(t,p)&&(c[p]=t[p]);c.originalType=e,c[m]="string"==typeof e?e:o,i[1]=c;for(var l=2;l<a;l++)i[l]=r[l];return n.createElement.apply(null,i)}return n.createElement.apply(null,r)}f.displayName="MDXCreateElement"},818:(e,t,r)=>{r.r(t),r.d(t,{assets:()=>p,contentTitle:()=>i,default:()=>u,frontMatter:()=>a,metadata:()=>c,toc:()=>l});var n=r(7462),o=(r(7294),r(3905));const a={id:"index",sidebar_position:1},i="Project Managament",c={unversionedId:"project-management/index",id:"project-management/index",title:"Project Managament",description:"We scrumming in our way ...",source:"@site/docs/project-management/index.md",sourceDirName:"project-management",slug:"/project-management/",permalink:"/docs/project-management/",draft:!1,editUrl:"https://github.com/foomo/foomo-docs/tree/main/foomo/docs/project-management/index.md",tags:[],version:"current",sidebarPosition:1,frontMatter:{id:"index",sidebar_position:1},sidebar:"pmSidebar",next:{title:"Responsibility",permalink:"/docs/project-management/responsibility"}},p={},l=[{value:"Values",id:"values",level:2},{value:"Epics",id:"epics",level:2}],s={toc:l},m="wrapper";function u(e){let{components:t,...r}=e;return(0,o.kt)(m,(0,n.Z)({},s,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"project-managament"},"Project Managament"),(0,o.kt)("p",null,"We ",(0,o.kt)("a",{parentName:"p",href:"https://www.scrum.org/"},"scrumming")," in our way ..."),(0,o.kt)("h2",{id:"values"},"Values"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"project managers manage projects not people")),(0,o.kt)("h2",{id:"epics"},"Epics"),(0,o.kt)("p",null,"Epics are the starting point ...."))}u.isMDXComponent=!0}}]);
|
||||
@ -1 +0,0 @@
|
||||
"use strict";(self.webpackChunkfoomo=self.webpackChunkfoomo||[]).push([[2302],{3905:(e,t,r)=>{r.d(t,{Zo:()=>l,kt:()=>y});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function i(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function a(e){for(var t=1;t<arguments.length;t++){var r=null!=arguments[t]?arguments[t]:{};t%2?i(Object(r),!0).forEach((function(t){o(e,t,r[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(r)):i(Object(r)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(r,t))}))}return e}function c(e,t){if(null==e)return{};var r,n,o=function(e,t){if(null==e)return{};var r,n,o={},i=Object.keys(e);for(n=0;n<i.length;n++)r=i[n],t.indexOf(r)>=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(n=0;n<i.length;n++)r=i[n],t.indexOf(r)>=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var p=n.createContext({}),s=function(e){var t=n.useContext(p),r=t;return e&&(r="function"==typeof e?e(t):a(a({},t),e)),r},l=function(e){var t=s(e.components);return n.createElement(p.Provider,{value:t},e.children)},m="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},f=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,i=e.originalType,p=e.parentName,l=c(e,["components","mdxType","originalType","parentName"]),m=s(r),f=o,y=m["".concat(p,".").concat(f)]||m[f]||u[f]||i;return r?n.createElement(y,a(a({ref:t},l),{},{components:r})):n.createElement(y,a({ref:t},l))}));function y(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=r.length,a=new Array(i);a[0]=f;var c={};for(var p in t)hasOwnProperty.call(t,p)&&(c[p]=t[p]);c.originalType=e,c[m]="string"==typeof e?e:o,a[1]=c;for(var s=2;s<i;s++)a[s]=r[s];return n.createElement.apply(null,a)}return n.createElement.apply(null,r)}f.displayName="MDXCreateElement"},80:(e,t,r)=>{r.r(t),r.d(t,{assets:()=>p,contentTitle:()=>a,default:()=>u,frontMatter:()=>i,metadata:()=>c,toc:()=>s});var n=r(7462),o=(r(7294),r(3905));const i={},a="Responsibility",c={unversionedId:"project-management/responsibility",id:"project-management/responsibility",title:"Responsibility",description:"Taking responsibility for a project ...",source:"@site/docs/project-management/responsibility.md",sourceDirName:"project-management",slug:"/project-management/responsibility",permalink:"/docs/project-management/responsibility",draft:!1,editUrl:"https://github.com/foomo/foomo-docs/tree/main/foomo/docs/project-management/responsibility.md",tags:[],version:"current",frontMatter:{},sidebar:"pmSidebar",previous:{title:"Project Managament",permalink:"/docs/project-management/"}},p={},s=[],l={toc:s},m="wrapper";function u(e){let{components:t,...r}=e;return(0,o.kt)(m,(0,n.Z)({},l,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"responsibility"},"Responsibility"),(0,o.kt)("p",null,"Taking responsibility for a project ..."))}u.isMDXComponent=!0}}]);
|
||||
1
assets/js/935f2afb.4d4134e4.js
Normal file
1
assets/js/935f2afb.4d4134e4.js
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
1
assets/js/9724.5ae4affb.js
Normal file
1
assets/js/9724.5ae4affb.js
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
1
assets/js/aedc5931.9e04ce9b.js
Normal file
1
assets/js/aedc5931.9e04ce9b.js
Normal file
File diff suppressed because one or more lines are too long
2
assets/js/main.0e9b6dcd.js
Normal file
2
assets/js/main.0e9b6dcd.js
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
1
assets/js/runtime~main.881e34e7.js
Normal file
1
assets/js/runtime~main.881e34e7.js
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -9,13 +9,13 @@
|
||||
|
||||
|
||||
<link rel="search" type="application/opensearchdescription+xml" title="foomo project docs" href="/opensearch.xml"><link rel="stylesheet" href="/assets/css/styles.45fe17b6.css">
|
||||
<link rel="preload" href="/assets/js/runtime~main.926d63f1.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.84c4b0c2.js" as="script">
|
||||
<link rel="preload" href="/assets/js/runtime~main.881e34e7.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.0e9b6dcd.js" as="script">
|
||||
</head>
|
||||
<body class="navigation-with-keyboard">
|
||||
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus">
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/project-management">PM</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><header class="hero hero--primary"><div class="container"><h1 class="hero__title">Archive</h1><p class="hero__subtitle">Archive</p></div></header><main><section class="margin-vert--lg"><div class="container"><div class="row"><div class="col col--4 margin-vert--lg"><h3>2021</h3><ul><li><a href="/blog/welcome-back-2021">November 12, 2021<!-- --> - <!-- -->Relaunching foomo.org</a></li></ul></div><div class="col col--4 margin-vert--lg"><h3>2022</h3><ul><li><a href="/blog/why-bundle-size-is-important">March 17, 2022<!-- --> - <!-- -->Why bundle size is important?</a></li><li><a href="/blog/prometheus-cardinality-issues">January 25, 2022<!-- --> - <!-- -->Prometheus Is Out Of Memory. Again.</a></li><li><a href="/blog/searching-for-search-engines">January 20, 2022<!-- --> - <!-- -->The never ending search a search engine 2022-01 edition</a></li><li><a href="/blog/impact-of-3rd-party-scripts-on-performance">January 20, 2022<!-- --> - <!-- -->Impact of 3rd party scripts on performance</a></li><li><a href="/blog/debugging-go-map-races-in-k8s">January 19, 2022<!-- --> - <!-- -->debugging Go map races in k8s</a></li></ul></div><div class="col col--4 margin-vert--lg"><h3>2023</h3><ul><li><a href="/blog/go-race-conditions-testing-and-coverage">March 17, 2023<!-- --> - <!-- -->Go race conditions testing and coverage</a></li><li><a href="/blog/accuracy-of-decimal-computations">March 6, 2023<!-- --> - <!-- -->Accuracy of decimal computations</a></li></ul></div></div></div></section></main></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.926d63f1.js"></script>
|
||||
<script src="/assets/js/main.84c4b0c2.js"></script>
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><header class="hero hero--primary"><div class="container"><h1 class="hero__title">Archive</h1><p class="hero__subtitle">Archive</p></div></header><main><section class="margin-vert--lg"><div class="container"><div class="row"><div class="col col--4 margin-vert--lg"><h3>2021</h3><ul><li><a href="/blog/welcome-back-2021">November 12, 2021<!-- --> - <!-- -->Relaunching foomo.org</a></li></ul></div><div class="col col--4 margin-vert--lg"><h3>2022</h3><ul><li><a href="/blog/why-bundle-size-is-important">March 17, 2022<!-- --> - <!-- -->Why bundle size is important?</a></li><li><a href="/blog/prometheus-cardinality-issues">January 25, 2022<!-- --> - <!-- -->Prometheus Is Out Of Memory. Again.</a></li><li><a href="/blog/searching-for-search-engines">January 20, 2022<!-- --> - <!-- -->The never ending search a search engine 2022-01 edition</a></li><li><a href="/blog/impact-of-3rd-party-scripts-on-performance">January 20, 2022<!-- --> - <!-- -->Impact of 3rd party scripts on performance</a></li><li><a href="/blog/debugging-go-map-races-in-k8s">January 19, 2022<!-- --> - <!-- -->debugging Go map races in k8s</a></li></ul></div><div class="col col--4 margin-vert--lg"><h3>2023</h3><ul><li><a href="/blog/go-race-conditions-testing-and-coverage">March 17, 2023<!-- --> - <!-- -->Go race conditions testing and coverage</a></li><li><a href="/blog/accuracy-of-decimal-computations">March 6, 2023<!-- --> - <!-- -->Accuracy of decimal computations</a></li></ul></div></div></div></section></main></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.881e34e7.js"></script>
|
||||
<script src="/assets/js/main.0e9b6dcd.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -9,12 +9,12 @@
|
||||
|
||||
|
||||
<link rel="search" type="application/opensearchdescription+xml" title="foomo project docs" href="/opensearch.xml"><link rel="stylesheet" href="/assets/css/styles.45fe17b6.css">
|
||||
<link rel="preload" href="/assets/js/runtime~main.926d63f1.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.84c4b0c2.js" as="script">
|
||||
<link rel="preload" href="/assets/js/runtime~main.881e34e7.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.0e9b6dcd.js" as="script">
|
||||
</head>
|
||||
<body class="navigation-with-keyboard">
|
||||
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus">
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/project-management">PM</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a aria-current="page" class="sidebarItemLink_mo7H sidebarItemLinkActive_I1ZP" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><article itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h1 class="title_f1Hy" itemprop="headline">Prometheus Is Out Of Memory. Again.</h1><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div id="post-content" class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a aria-current="page" class="sidebarItemLink_mo7H sidebarItemLinkActive_I1ZP" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><article itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h1 class="title_f1Hy" itemprop="headline">Prometheus Is Out Of Memory. Again.</h1><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div id="post-content" class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
Prometheus returns us a 503, a trusty way of saying I'm not ready, and I'm probably going to die soon.
|
||||
And since we're running in kubernetes I'm going to die soon, again and again.
|
||||
And you're getting reports from your colleagues that prometheus is not responding.
|
||||
@ -34,7 +34,7 @@ Or to quote user <a href="https://www.reddit.com/user/naveen17797/" target="_bla
|
||||
Since we can't get prometheus up and running to utilize PromQL to detect the potential issues, we have to find another way to detect high cardinality.
|
||||
Therefore, we might want to get our hands dirty with some <code>kubectl exec -it -n monitoring pods/prometheus-prometheus-kube-prometheus-prometheus-0 -- sh</code>, and run the prometheus <code>tsdb</code> analysis too.</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">/prometheus $ promtool tsdb analyze </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">.</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Which produced the result.</p><div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Block ID: 01FT8E8YY4THHZ2S7C3G04GJMG</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Duration: 1h59m59.997s</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Series: 564171</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Label names: 285</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings (unique label pairs): 21139</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings entries (total label pairs): 6423664</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Highest cardinality metric names:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> 11340 haproxy_server_http_responses_total</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>We see the potential issue here, where the <code>haproxy_server_http_responses_total</code> metric is having a super-high cardinality which is growing.
|
||||
We need to deal with it, so that our prometheus instance can breathe again. In this particular case, the solution was updating the haproxy.</p><p>... or burn it, up to you.</p><p><img loading="lazy" alt="Flame Thrower" src="/assets/images/flame-thrower-56bcf89132356ff53c03ca029d9d0746.webp" width="1440" height="1080" class="img_ev3q"></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-further-reading">The Further Reading<a href="#the-further-reading" class="hash-link" aria-label="Direct link to The Further Reading" title="Direct link to The Further Reading"></a></h2><ol><li><a href="https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/wal.md" target="_blank" rel="noopener noreferrer">WAL Definition</a></li><li><a href="https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/" target="_blank" rel="noopener noreferrer">WAL & Checkpoints</a></li><li><a href="https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality" target="_blank" rel="noopener noreferrer">Using TSDB</a></li><li><a href="https://www.robustperception.io/which-are-my-biggest-metrics" target="_blank" rel="noopener noreferrer">Biggest Metrics</a></li><li><a href="https://www.robustperception.io/cardinality-is-key" target="_blank" rel="noopener noreferrer">Cardinality</a></li></ol></div><footer class="row docusaurus-mt-lg blogPostFooterDetailsFull_mRVl"><div class="col"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/prometheus">prometheus</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/cardinality">cardinality</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/devops">devops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/ops">ops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/k-8-s">k8s</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/oom">oom</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/memory">memory</a></li></ul></div><div class="col margin-top--sm"><a href="https://github.com/foomo/foomo-docs/tree/main/foomo/blog/2022-01-25-prometheus-cardinality-issues/index.mdx" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_Z9Sw" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>Edit this page</a></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Blog post page navigation"><a class="pagination-nav__link pagination-nav__link--prev" href="/blog/why-bundle-size-is-important"><div class="pagination-nav__sublabel">Newer Post</div><div class="pagination-nav__label">Why bundle size is important?</div></a><a class="pagination-nav__link pagination-nav__link--next" href="/blog/searching-for-search-engines"><div class="pagination-nav__sublabel">Older Post</div><div class="pagination-nav__label">The never ending search a search engine 2022-01 edition</div></a></nav></main><div class="col col--2"><div class="tableOfContents_bqdL thin-scrollbar"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#the-annoyance" class="table-of-contents__link toc-highlight">The Annoyance</a></li><li><a href="#the-problem" class="table-of-contents__link toc-highlight">The Problem</a></li><li><a href="#the-source" class="table-of-contents__link toc-highlight">The Source</a></li><li><a href="#the-solution" class="table-of-contents__link toc-highlight">The Solution</a></li><li><a href="#the-further-reading" class="table-of-contents__link toc-highlight">The Further Reading</a></li></ul></div></div></div></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.926d63f1.js"></script>
|
||||
<script src="/assets/js/main.84c4b0c2.js"></script>
|
||||
<script src="/assets/js/runtime~main.881e34e7.js"></script>
|
||||
<script src="/assets/js/main.0e9b6dcd.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -9,12 +9,12 @@
|
||||
|
||||
|
||||
<link rel="search" type="application/opensearchdescription+xml" title="foomo project docs" href="/opensearch.xml"><link rel="stylesheet" href="/assets/css/styles.45fe17b6.css">
|
||||
<link rel="preload" href="/assets/js/runtime~main.926d63f1.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.84c4b0c2.js" as="script">
|
||||
<link rel="preload" href="/assets/js/runtime~main.881e34e7.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.0e9b6dcd.js" as="script">
|
||||
</head>
|
||||
<body class="navigation-with-keyboard">
|
||||
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus">
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/project-management">PM</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "cardinality"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "cardinality"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
Prometheus returns us a 503, a trusty way of saying I'm not ready, and I'm probably going to die soon.
|
||||
And since we're running in kubernetes I'm going to die soon, again and again.
|
||||
And you're getting reports from your colleagues that prometheus is not responding.
|
||||
@ -34,7 +34,7 @@ Or to quote user <a href="https://www.reddit.com/user/naveen17797/" target="_bla
|
||||
Since we can't get prometheus up and running to utilize PromQL to detect the potential issues, we have to find another way to detect high cardinality.
|
||||
Therefore, we might want to get our hands dirty with some <code>kubectl exec -it -n monitoring pods/prometheus-prometheus-kube-prometheus-prometheus-0 -- sh</code>, and run the prometheus <code>tsdb</code> analysis too.</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">/prometheus $ promtool tsdb analyze </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">.</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Which produced the result.</p><div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Block ID: 01FT8E8YY4THHZ2S7C3G04GJMG</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Duration: 1h59m59.997s</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Series: 564171</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Label names: 285</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings (unique label pairs): 21139</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings entries (total label pairs): 6423664</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Highest cardinality metric names:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> 11340 haproxy_server_http_responses_total</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>We see the potential issue here, where the <code>haproxy_server_http_responses_total</code> metric is having a super-high cardinality which is growing.
|
||||
We need to deal with it, so that our prometheus instance can breathe again. In this particular case, the solution was updating the haproxy.</p><p>... or burn it, up to you.</p><p><img loading="lazy" alt="Flame Thrower" src="/assets/images/flame-thrower-56bcf89132356ff53c03ca029d9d0746.webp" width="1440" height="1080" class="img_ev3q"></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-further-reading">The Further Reading<a href="#the-further-reading" class="hash-link" aria-label="Direct link to The Further Reading" title="Direct link to The Further Reading"></a></h2><ol><li><a href="https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/wal.md" target="_blank" rel="noopener noreferrer">WAL Definition</a></li><li><a href="https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/" target="_blank" rel="noopener noreferrer">WAL & Checkpoints</a></li><li><a href="https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality" target="_blank" rel="noopener noreferrer">Using TSDB</a></li><li><a href="https://www.robustperception.io/which-are-my-biggest-metrics" target="_blank" rel="noopener noreferrer">Biggest Metrics</a></li><li><a href="https://www.robustperception.io/cardinality-is-key" target="_blank" rel="noopener noreferrer">Cardinality</a></li></ol></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/prometheus">prometheus</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/cardinality">cardinality</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/devops">devops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/ops">ops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/k-8-s">k8s</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/oom">oom</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/memory">memory</a></li></ul></div></footer></article><nav class="pagination-nav" aria-label="Blog list page navigation"></nav></main></div></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.926d63f1.js"></script>
|
||||
<script src="/assets/js/main.84c4b0c2.js"></script>
|
||||
<script src="/assets/js/runtime~main.881e34e7.js"></script>
|
||||
<script src="/assets/js/main.0e9b6dcd.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -9,12 +9,12 @@
|
||||
|
||||
|
||||
<link rel="search" type="application/opensearchdescription+xml" title="foomo project docs" href="/opensearch.xml"><link rel="stylesheet" href="/assets/css/styles.45fe17b6.css">
|
||||
<link rel="preload" href="/assets/js/runtime~main.926d63f1.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.84c4b0c2.js" as="script">
|
||||
<link rel="preload" href="/assets/js/runtime~main.881e34e7.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.0e9b6dcd.js" as="script">
|
||||
</head>
|
||||
<body class="navigation-with-keyboard">
|
||||
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus">
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/project-management">PM</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "devops"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "devops"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
Prometheus returns us a 503, a trusty way of saying I'm not ready, and I'm probably going to die soon.
|
||||
And since we're running in kubernetes I'm going to die soon, again and again.
|
||||
And you're getting reports from your colleagues that prometheus is not responding.
|
||||
@ -34,7 +34,7 @@ Or to quote user <a href="https://www.reddit.com/user/naveen17797/" target="_bla
|
||||
Since we can't get prometheus up and running to utilize PromQL to detect the potential issues, we have to find another way to detect high cardinality.
|
||||
Therefore, we might want to get our hands dirty with some <code>kubectl exec -it -n monitoring pods/prometheus-prometheus-kube-prometheus-prometheus-0 -- sh</code>, and run the prometheus <code>tsdb</code> analysis too.</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">/prometheus $ promtool tsdb analyze </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">.</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Which produced the result.</p><div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Block ID: 01FT8E8YY4THHZ2S7C3G04GJMG</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Duration: 1h59m59.997s</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Series: 564171</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Label names: 285</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings (unique label pairs): 21139</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings entries (total label pairs): 6423664</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Highest cardinality metric names:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> 11340 haproxy_server_http_responses_total</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>We see the potential issue here, where the <code>haproxy_server_http_responses_total</code> metric is having a super-high cardinality which is growing.
|
||||
We need to deal with it, so that our prometheus instance can breathe again. In this particular case, the solution was updating the haproxy.</p><p>... or burn it, up to you.</p><p><img loading="lazy" alt="Flame Thrower" src="/assets/images/flame-thrower-56bcf89132356ff53c03ca029d9d0746.webp" width="1440" height="1080" class="img_ev3q"></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-further-reading">The Further Reading<a href="#the-further-reading" class="hash-link" aria-label="Direct link to The Further Reading" title="Direct link to The Further Reading"></a></h2><ol><li><a href="https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/wal.md" target="_blank" rel="noopener noreferrer">WAL Definition</a></li><li><a href="https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/" target="_blank" rel="noopener noreferrer">WAL & Checkpoints</a></li><li><a href="https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality" target="_blank" rel="noopener noreferrer">Using TSDB</a></li><li><a href="https://www.robustperception.io/which-are-my-biggest-metrics" target="_blank" rel="noopener noreferrer">Biggest Metrics</a></li><li><a href="https://www.robustperception.io/cardinality-is-key" target="_blank" rel="noopener noreferrer">Cardinality</a></li></ol></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/prometheus">prometheus</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/cardinality">cardinality</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/devops">devops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/ops">ops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/k-8-s">k8s</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/oom">oom</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/memory">memory</a></li></ul></div></footer></article><nav class="pagination-nav" aria-label="Blog list page navigation"></nav></main></div></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.926d63f1.js"></script>
|
||||
<script src="/assets/js/main.84c4b0c2.js"></script>
|
||||
<script src="/assets/js/runtime~main.881e34e7.js"></script>
|
||||
<script src="/assets/js/main.0e9b6dcd.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -9,12 +9,12 @@
|
||||
|
||||
|
||||
<link rel="search" type="application/opensearchdescription+xml" title="foomo project docs" href="/opensearch.xml"><link rel="stylesheet" href="/assets/css/styles.45fe17b6.css">
|
||||
<link rel="preload" href="/assets/js/runtime~main.926d63f1.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.84c4b0c2.js" as="script">
|
||||
<link rel="preload" href="/assets/js/runtime~main.881e34e7.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.0e9b6dcd.js" as="script">
|
||||
</head>
|
||||
<body class="navigation-with-keyboard">
|
||||
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus">
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/project-management">PM</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "k8s"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "k8s"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
Prometheus returns us a 503, a trusty way of saying I'm not ready, and I'm probably going to die soon.
|
||||
And since we're running in kubernetes I'm going to die soon, again and again.
|
||||
And you're getting reports from your colleagues that prometheus is not responding.
|
||||
@ -34,7 +34,7 @@ Or to quote user <a href="https://www.reddit.com/user/naveen17797/" target="_bla
|
||||
Since we can't get prometheus up and running to utilize PromQL to detect the potential issues, we have to find another way to detect high cardinality.
|
||||
Therefore, we might want to get our hands dirty with some <code>kubectl exec -it -n monitoring pods/prometheus-prometheus-kube-prometheus-prometheus-0 -- sh</code>, and run the prometheus <code>tsdb</code> analysis too.</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">/prometheus $ promtool tsdb analyze </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">.</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Which produced the result.</p><div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Block ID: 01FT8E8YY4THHZ2S7C3G04GJMG</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Duration: 1h59m59.997s</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Series: 564171</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Label names: 285</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings (unique label pairs): 21139</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings entries (total label pairs): 6423664</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Highest cardinality metric names:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> 11340 haproxy_server_http_responses_total</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>We see the potential issue here, where the <code>haproxy_server_http_responses_total</code> metric is having a super-high cardinality which is growing.
|
||||
We need to deal with it, so that our prometheus instance can breathe again. In this particular case, the solution was updating the haproxy.</p><p>... or burn it, up to you.</p><p><img loading="lazy" alt="Flame Thrower" src="/assets/images/flame-thrower-56bcf89132356ff53c03ca029d9d0746.webp" width="1440" height="1080" class="img_ev3q"></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-further-reading">The Further Reading<a href="#the-further-reading" class="hash-link" aria-label="Direct link to The Further Reading" title="Direct link to The Further Reading"></a></h2><ol><li><a href="https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/wal.md" target="_blank" rel="noopener noreferrer">WAL Definition</a></li><li><a href="https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/" target="_blank" rel="noopener noreferrer">WAL & Checkpoints</a></li><li><a href="https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality" target="_blank" rel="noopener noreferrer">Using TSDB</a></li><li><a href="https://www.robustperception.io/which-are-my-biggest-metrics" target="_blank" rel="noopener noreferrer">Biggest Metrics</a></li><li><a href="https://www.robustperception.io/cardinality-is-key" target="_blank" rel="noopener noreferrer">Cardinality</a></li></ol></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/prometheus">prometheus</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/cardinality">cardinality</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/devops">devops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/ops">ops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/k-8-s">k8s</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/oom">oom</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/memory">memory</a></li></ul></div></footer></article><nav class="pagination-nav" aria-label="Blog list page navigation"></nav></main></div></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.926d63f1.js"></script>
|
||||
<script src="/assets/js/main.84c4b0c2.js"></script>
|
||||
<script src="/assets/js/runtime~main.881e34e7.js"></script>
|
||||
<script src="/assets/js/main.0e9b6dcd.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@ -9,12 +9,12 @@
|
||||
|
||||
|
||||
<link rel="search" type="application/opensearchdescription+xml" title="foomo project docs" href="/opensearch.xml"><link rel="stylesheet" href="/assets/css/styles.45fe17b6.css">
|
||||
<link rel="preload" href="/assets/js/runtime~main.926d63f1.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.84c4b0c2.js" as="script">
|
||||
<link rel="preload" href="/assets/js/runtime~main.881e34e7.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.0e9b6dcd.js" as="script">
|
||||
</head>
|
||||
<body class="navigation-with-keyboard">
|
||||
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus">
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/project-management">PM</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "memory"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "memory"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
Prometheus returns us a 503, a trusty way of saying I'm not ready, and I'm probably going to die soon.
|
||||
And since we're running in kubernetes I'm going to die soon, again and again.
|
||||
And you're getting reports from your colleagues that prometheus is not responding.
|
||||
@ -34,7 +34,7 @@ Or to quote user <a href="https://www.reddit.com/user/naveen17797/" target="_bla
|
||||
Since we can't get prometheus up and running to utilize PromQL to detect the potential issues, we have to find another way to detect high cardinality.
|
||||
Therefore, we might want to get our hands dirty with some <code>kubectl exec -it -n monitoring pods/prometheus-prometheus-kube-prometheus-prometheus-0 -- sh</code>, and run the prometheus <code>tsdb</code> analysis too.</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">/prometheus $ promtool tsdb analyze </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">.</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Which produced the result.</p><div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Block ID: 01FT8E8YY4THHZ2S7C3G04GJMG</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Duration: 1h59m59.997s</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Series: 564171</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Label names: 285</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings (unique label pairs): 21139</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings entries (total label pairs): 6423664</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Highest cardinality metric names:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> 11340 haproxy_server_http_responses_total</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>We see the potential issue here, where the <code>haproxy_server_http_responses_total</code> metric is having a super-high cardinality which is growing.
|
||||
We need to deal with it, so that our prometheus instance can breathe again. In this particular case, the solution was updating the haproxy.</p><p>... or burn it, up to you.</p><p><img loading="lazy" alt="Flame Thrower" src="/assets/images/flame-thrower-56bcf89132356ff53c03ca029d9d0746.webp" width="1440" height="1080" class="img_ev3q"></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-further-reading">The Further Reading<a href="#the-further-reading" class="hash-link" aria-label="Direct link to The Further Reading" title="Direct link to The Further Reading"></a></h2><ol><li><a href="https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/wal.md" target="_blank" rel="noopener noreferrer">WAL Definition</a></li><li><a href="https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/" target="_blank" rel="noopener noreferrer">WAL & Checkpoints</a></li><li><a href="https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality" target="_blank" rel="noopener noreferrer">Using TSDB</a></li><li><a href="https://www.robustperception.io/which-are-my-biggest-metrics" target="_blank" rel="noopener noreferrer">Biggest Metrics</a></li><li><a href="https://www.robustperception.io/cardinality-is-key" target="_blank" rel="noopener noreferrer">Cardinality</a></li></ol></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/prometheus">prometheus</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/cardinality">cardinality</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/devops">devops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/ops">ops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/k-8-s">k8s</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/oom">oom</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/memory">memory</a></li></ul></div></footer></article><nav class="pagination-nav" aria-label="Blog list page navigation"></nav></main></div></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.926d63f1.js"></script>
|
||||
<script src="/assets/js/main.84c4b0c2.js"></script>
|
||||
<script src="/assets/js/runtime~main.881e34e7.js"></script>
|
||||
<script src="/assets/js/main.0e9b6dcd.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@ -9,12 +9,12 @@
|
||||
|
||||
|
||||
<link rel="search" type="application/opensearchdescription+xml" title="foomo project docs" href="/opensearch.xml"><link rel="stylesheet" href="/assets/css/styles.45fe17b6.css">
|
||||
<link rel="preload" href="/assets/js/runtime~main.926d63f1.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.84c4b0c2.js" as="script">
|
||||
<link rel="preload" href="/assets/js/runtime~main.881e34e7.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.0e9b6dcd.js" as="script">
|
||||
</head>
|
||||
<body class="navigation-with-keyboard">
|
||||
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus">
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/project-management">PM</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "oom"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "oom"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
Prometheus returns us a 503, a trusty way of saying I'm not ready, and I'm probably going to die soon.
|
||||
And since we're running in kubernetes I'm going to die soon, again and again.
|
||||
And you're getting reports from your colleagues that prometheus is not responding.
|
||||
@ -34,7 +34,7 @@ Or to quote user <a href="https://www.reddit.com/user/naveen17797/" target="_bla
|
||||
Since we can't get prometheus up and running to utilize PromQL to detect the potential issues, we have to find another way to detect high cardinality.
|
||||
Therefore, we might want to get our hands dirty with some <code>kubectl exec -it -n monitoring pods/prometheus-prometheus-kube-prometheus-prometheus-0 -- sh</code>, and run the prometheus <code>tsdb</code> analysis too.</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">/prometheus $ promtool tsdb analyze </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">.</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Which produced the result.</p><div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Block ID: 01FT8E8YY4THHZ2S7C3G04GJMG</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Duration: 1h59m59.997s</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Series: 564171</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Label names: 285</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings (unique label pairs): 21139</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings entries (total label pairs): 6423664</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Highest cardinality metric names:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> 11340 haproxy_server_http_responses_total</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>We see the potential issue here, where the <code>haproxy_server_http_responses_total</code> metric is having a super-high cardinality which is growing.
|
||||
We need to deal with it, so that our prometheus instance can breathe again. In this particular case, the solution was updating the haproxy.</p><p>... or burn it, up to you.</p><p><img loading="lazy" alt="Flame Thrower" src="/assets/images/flame-thrower-56bcf89132356ff53c03ca029d9d0746.webp" width="1440" height="1080" class="img_ev3q"></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-further-reading">The Further Reading<a href="#the-further-reading" class="hash-link" aria-label="Direct link to The Further Reading" title="Direct link to The Further Reading"></a></h2><ol><li><a href="https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/wal.md" target="_blank" rel="noopener noreferrer">WAL Definition</a></li><li><a href="https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/" target="_blank" rel="noopener noreferrer">WAL & Checkpoints</a></li><li><a href="https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality" target="_blank" rel="noopener noreferrer">Using TSDB</a></li><li><a href="https://www.robustperception.io/which-are-my-biggest-metrics" target="_blank" rel="noopener noreferrer">Biggest Metrics</a></li><li><a href="https://www.robustperception.io/cardinality-is-key" target="_blank" rel="noopener noreferrer">Cardinality</a></li></ol></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/prometheus">prometheus</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/cardinality">cardinality</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/devops">devops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/ops">ops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/k-8-s">k8s</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/oom">oom</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/memory">memory</a></li></ul></div></footer></article><nav class="pagination-nav" aria-label="Blog list page navigation"></nav></main></div></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.926d63f1.js"></script>
|
||||
<script src="/assets/js/main.84c4b0c2.js"></script>
|
||||
<script src="/assets/js/runtime~main.881e34e7.js"></script>
|
||||
<script src="/assets/js/main.0e9b6dcd.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@ -9,12 +9,12 @@
|
||||
|
||||
|
||||
<link rel="search" type="application/opensearchdescription+xml" title="foomo project docs" href="/opensearch.xml"><link rel="stylesheet" href="/assets/css/styles.45fe17b6.css">
|
||||
<link rel="preload" href="/assets/js/runtime~main.926d63f1.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.84c4b0c2.js" as="script">
|
||||
<link rel="preload" href="/assets/js/runtime~main.881e34e7.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.0e9b6dcd.js" as="script">
|
||||
</head>
|
||||
<body class="navigation-with-keyboard">
|
||||
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus">
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/project-management">PM</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "ops"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "ops"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
Prometheus returns us a 503, a trusty way of saying I'm not ready, and I'm probably going to die soon.
|
||||
And since we're running in kubernetes I'm going to die soon, again and again.
|
||||
And you're getting reports from your colleagues that prometheus is not responding.
|
||||
@ -34,7 +34,7 @@ Or to quote user <a href="https://www.reddit.com/user/naveen17797/" target="_bla
|
||||
Since we can't get prometheus up and running to utilize PromQL to detect the potential issues, we have to find another way to detect high cardinality.
|
||||
Therefore, we might want to get our hands dirty with some <code>kubectl exec -it -n monitoring pods/prometheus-prometheus-kube-prometheus-prometheus-0 -- sh</code>, and run the prometheus <code>tsdb</code> analysis too.</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">/prometheus $ promtool tsdb analyze </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">.</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Which produced the result.</p><div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Block ID: 01FT8E8YY4THHZ2S7C3G04GJMG</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Duration: 1h59m59.997s</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Series: 564171</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Label names: 285</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings (unique label pairs): 21139</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings entries (total label pairs): 6423664</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Highest cardinality metric names:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> 11340 haproxy_server_http_responses_total</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>We see the potential issue here, where the <code>haproxy_server_http_responses_total</code> metric is having a super-high cardinality which is growing.
|
||||
We need to deal with it, so that our prometheus instance can breathe again. In this particular case, the solution was updating the haproxy.</p><p>... or burn it, up to you.</p><p><img loading="lazy" alt="Flame Thrower" src="/assets/images/flame-thrower-56bcf89132356ff53c03ca029d9d0746.webp" width="1440" height="1080" class="img_ev3q"></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-further-reading">The Further Reading<a href="#the-further-reading" class="hash-link" aria-label="Direct link to The Further Reading" title="Direct link to The Further Reading"></a></h2><ol><li><a href="https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/wal.md" target="_blank" rel="noopener noreferrer">WAL Definition</a></li><li><a href="https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/" target="_blank" rel="noopener noreferrer">WAL & Checkpoints</a></li><li><a href="https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality" target="_blank" rel="noopener noreferrer">Using TSDB</a></li><li><a href="https://www.robustperception.io/which-are-my-biggest-metrics" target="_blank" rel="noopener noreferrer">Biggest Metrics</a></li><li><a href="https://www.robustperception.io/cardinality-is-key" target="_blank" rel="noopener noreferrer">Cardinality</a></li></ol></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/prometheus">prometheus</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/cardinality">cardinality</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/devops">devops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/ops">ops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/k-8-s">k8s</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/oom">oom</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/memory">memory</a></li></ul></div></footer></article><nav class="pagination-nav" aria-label="Blog list page navigation"></nav></main></div></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.926d63f1.js"></script>
|
||||
<script src="/assets/js/main.84c4b0c2.js"></script>
|
||||
<script src="/assets/js/runtime~main.881e34e7.js"></script>
|
||||
<script src="/assets/js/main.0e9b6dcd.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because one or more lines are too long
@ -9,12 +9,12 @@
|
||||
|
||||
|
||||
<link rel="search" type="application/opensearchdescription+xml" title="foomo project docs" href="/opensearch.xml"><link rel="stylesheet" href="/assets/css/styles.45fe17b6.css">
|
||||
<link rel="preload" href="/assets/js/runtime~main.926d63f1.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.84c4b0c2.js" as="script">
|
||||
<link rel="preload" href="/assets/js/runtime~main.881e34e7.js" as="script">
|
||||
<link rel="preload" href="/assets/js/main.0e9b6dcd.js" as="script">
|
||||
</head>
|
||||
<body class="navigation-with-keyboard">
|
||||
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus">
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/project-management">PM</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "prometheus"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
<div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title text--truncate">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general">General</a><a class="navbar__item navbar__link" href="/docs/frontend">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend">Backend</a><a class="navbar__item navbar__link" href="/docs/devops">DevOps</a><a class="navbar__item navbar__link" href="/docs/projects">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">Recent posts</div><ul class="sidebarItemList_Yudw clean-list"><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/go-race-conditions-testing-and-coverage">Go race conditions testing and coverage</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/accuracy-of-decimal-computations">Accuracy of decimal computations</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/why-bundle-size-is-important">Why bundle size is important?</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem__DBe"><a class="sidebarItemLink_mo7H" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><header class="margin-bottom--xl"><h1>One post tagged with "prometheus"</h1><a href="/blog/tags">View All Tags</a></header><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="title_f1Hy" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="container_mt6G margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_Hf19"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link"><img class="avatar__photo" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-annoyance">The Annoyance<a href="#the-annoyance" class="hash-link" aria-label="Direct link to The Annoyance" title="Direct link to The Annoyance"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||||
Prometheus returns us a 503, a trusty way of saying I'm not ready, and I'm probably going to die soon.
|
||||
And since we're running in kubernetes I'm going to die soon, again and again.
|
||||
And you're getting reports from your colleagues that prometheus is not responding.
|
||||
@ -34,7 +34,7 @@ Or to quote user <a href="https://www.reddit.com/user/naveen17797/" target="_bla
|
||||
Since we can't get prometheus up and running to utilize PromQL to detect the potential issues, we have to find another way to detect high cardinality.
|
||||
Therefore, we might want to get our hands dirty with some <code>kubectl exec -it -n monitoring pods/prometheus-prometheus-kube-prometheus-prometheus-0 -- sh</code>, and run the prometheus <code>tsdb</code> analysis too.</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">/prometheus $ promtool tsdb analyze </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">.</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Which produced the result.</p><div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Block ID: 01FT8E8YY4THHZ2S7C3G04GJMG</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Duration: 1h59m59.997s</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Series: 564171</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Label names: 285</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings (unique label pairs): 21139</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings entries (total label pairs): 6423664</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Highest cardinality metric names:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> 11340 haproxy_server_http_responses_total</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>We see the potential issue here, where the <code>haproxy_server_http_responses_total</code> metric is having a super-high cardinality which is growing.
|
||||
We need to deal with it, so that our prometheus instance can breathe again. In this particular case, the solution was updating the haproxy.</p><p>... or burn it, up to you.</p><p><img loading="lazy" alt="Flame Thrower" src="/assets/images/flame-thrower-56bcf89132356ff53c03ca029d9d0746.webp" width="1440" height="1080" class="img_ev3q"></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="the-further-reading">The Further Reading<a href="#the-further-reading" class="hash-link" aria-label="Direct link to The Further Reading" title="Direct link to The Further Reading"></a></h2><ol><li><a href="https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/wal.md" target="_blank" rel="noopener noreferrer">WAL Definition</a></li><li><a href="https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/" target="_blank" rel="noopener noreferrer">WAL & Checkpoints</a></li><li><a href="https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality" target="_blank" rel="noopener noreferrer">Using TSDB</a></li><li><a href="https://www.robustperception.io/which-are-my-biggest-metrics" target="_blank" rel="noopener noreferrer">Biggest Metrics</a></li><li><a href="https://www.robustperception.io/cardinality-is-key" target="_blank" rel="noopener noreferrer">Cardinality</a></li></ol></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/prometheus">prometheus</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/cardinality">cardinality</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/devops">devops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/ops">ops</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/k-8-s">k8s</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/oom">oom</a></li><li class="tag_QGVx"><a class="tag_zVej tagRegular_sFm0" href="/blog/tags/memory">memory</a></li></ul></div></footer></article><nav class="pagination-nav" aria-label="Blog list page navigation"></nav></main></div></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">github</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/foomo" target="_blank" rel="noopener noreferrer" class="footer__link-item">https://github.com/foomo</a></li></ul></div><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2023 bestbytes</div></div></div></footer></div>
|
||||
<script src="/assets/js/runtime~main.926d63f1.js"></script>
|
||||
<script src="/assets/js/main.84c4b0c2.js"></script>
|
||||
<script src="/assets/js/runtime~main.881e34e7.js"></script>
|
||||
<script src="/assets/js/main.0e9b6dcd.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user