mirror of
https://github.com/foomo/foomo-docs.git
synced 2025-10-16 12:35:40 +00:00
41 lines
51 KiB
HTML
41 lines
51 KiB
HTML
<!doctype html>
|
||
<html lang="en" dir="ltr">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||
<meta name="generator" content="Docusaurus v2.0.0-beta.14">
|
||
<link rel="alternate" type="application/rss+xml" href="/blog/rss.xml" title="foomo project docs RSS Feed">
|
||
<link rel="alternate" type="application/atom+xml" href="/blog/atom.xml" title="foomo project docs Atom Feed">
|
||
<link rel="search" type="application/opensearchdescription+xml" title="foomo project docs" href="/opensearch.xml"><title data-react-helmet="true">Blog | foomo project docs</title><meta data-react-helmet="true" property="og:title" content="Blog | foomo project docs"><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"><meta data-react-helmet="true" name="description" content="Blog"><meta data-react-helmet="true" property="og:description" content="Blog"><meta data-react-helmet="true" property="og:url" content="https://www.foomo.org/blog"><meta data-react-helmet="true" name="docsearch:language" content="en"><meta data-react-helmet="true" name="docsearch:docusaurus_tag" content="blog_posts_list"><link data-react-helmet="true" rel="icon" href="/img/favicon.ico"><link data-react-helmet="true" rel="canonical" href="https://www.foomo.org/blog"><link data-react-helmet="true" rel="alternate" href="https://www.foomo.org/blog" hreflang="en"><link data-react-helmet="true" rel="alternate" href="https://www.foomo.org/blog" hreflang="x-default"><link data-react-helmet="true" rel="preconnect" href="https://SUATUVZDDM-dsn.algolia.net" crossorigin="anonymous"><link rel="stylesheet" href="/assets/css/styles.47738ace.css">
|
||
<link rel="preload" href="/assets/js/runtime~main.3b4333f9.js" as="script">
|
||
<link rel="preload" href="/assets/js/main.681e1e0c.js" as="script">
|
||
</head>
|
||
<body>
|
||
<script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus">
|
||
<div><a href="#" class="skipToContent_OuoZ">Skip to main content</a></div><nav class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Navigation bar toggle" class="navbar__toggle clean-btn" type="button" tabindex="0"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><b class="navbar__title">foomo</b></a><a class="navbar__item navbar__link" href="/docs/general/intro">General</a><a class="navbar__item navbar__link" href="/docs/frontend/intro">Frontend</a><a class="navbar__item navbar__link" href="/docs/backend/intro">Backend</a><a class="navbar__item navbar__link" href="/docs/devops/intro">DevOps</a><a class="navbar__item navbar__link" href="/docs/project-management/intro">PM</a><a class="navbar__item navbar__link" href="/docs/projects/intro">Projects</a></div><div class="navbar__items navbar__items--right"><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/blog">Blog</a><div class="searchBox_Utm0"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div class="main-wrapper blog-wrapper blog-list-page"><div class="container margin-vert--lg"><div class="row"><aside class="col col--3"><nav class="sidebar_q+wC thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_9G5K margin-bottom--md">Recent posts</div><ul class="sidebarItemList_6T4b"><li class="sidebarItem_cjdF"><a class="sidebarItemLink_zyXk" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></li><li class="sidebarItem_cjdF"><a class="sidebarItemLink_zyXk" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></li><li class="sidebarItem_cjdF"><a class="sidebarItemLink_zyXk" href="/blog/impact-of-3rd-party-scripts-on-performance">Impact of 3rd party scripts on performance</a></li><li class="sidebarItem_cjdF"><a class="sidebarItemLink_zyXk" href="/blog/debugging-go-map-races-in-k8s">debugging Go map races in k8s</a></li><li class="sidebarItem_cjdF"><a class="sidebarItemLink_zyXk" href="/blog/welcome-back-2021">Relaunching foomo.org</a></li></ul></nav></aside><main class="col col--7" itemscope="" itemtype="http://schema.org/Blog"><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="blogPostTitle_d4p0" itemprop="headline"><a itemprop="url" href="/blog/prometheus-cardinality-issues">Prometheus Is Out Of Memory. Again.</a></h2><div class="blogPostData_-Im+ margin-vert--md"><time datetime="2022-01-25T00:00:00.000Z" itemprop="datePublished">January 25, 2022</time></div><div class="row margin-top--md margin-bottom--sm"><div class="col col--6 authorCol_8c0z"><div class="avatar margin-bottom--sm"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" class="avatar__photo-link avatar__photo"><img class="image_9q7L" src="https://github.com/smartinov.png" alt="Stefan Martinov"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/smartinov" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Stefan Martinov</span></a></div><small class="avatar__subtitle" itemprop="description">Memelord</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_y2LR" id="the-annoyance">The Annoyance<a class="hash-link" href="#the-annoyance" title="Direct link to heading"></a></h2><p>So, we've all been there. You go to your trusty grafana, search for some sweet metrics that you implemented and WHAM!
|
||
Prometheus returns us a 503, a trusty way of saying I'm not ready, and I'm probably going to die soon.
|
||
And since we're running in kubernetes I'm going to die soon, again and again.
|
||
And you're getting reports from your colleagues that prometheus is not responding.
|
||
And you can't ignore them anymore.</p><p><img alt="Bummer." src="/assets/images/bummer-e80d471cba23d1ee83e8463187845893.webp"></p><h2 class="anchor anchorWithStickyNavbar_y2LR" id="the-problem">The Problem<a class="hash-link" href="#the-problem" title="Direct link to heading"></a></h2><p>All right, lets check what's happening to the little guy.</p><div class="codeBlockContainer_J+bg language-bash theme-code-block"><div class="codeBlockContent_csEI bash"><pre tabindex="0" class="prism-code language-bash codeBlock_rtdJ thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl get pods -n monitoring</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">prometheus-prometheus-kube-prometheus-prometheus-0 </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain">/2 Running </span><span class="token number" style="color:rgb(247, 140, 108)">4</span><span class="token plain"> 5m</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>It seems like it's stuck in the running state, where the container is not yet ready.
|
||
Let's describe the deployment, to check out what's happening.</p><div class="codeBlockContainer_J+bg language-yaml theme-code-block"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token key atrule">State</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Running │</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token key atrule">Started</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Wed</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> 12 Jan 2022 15</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">12</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">49 +0100 │</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token key atrule">Last State</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Terminated │</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token key atrule">Reason</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> OOMKilled │</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token key atrule">Exit Code</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> 137 │</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token key atrule">Started</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Tue</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> 11 Jan 2022 17</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">14</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">41 +0100 │</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token key atrule">Finished</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Wed</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> 12 Jan 2022 15</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">12</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">47 +0100 │</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>So we see that the prometheus is in a running state waiting for the readiness probe to trigger, probably working on recovering from Write Ahead Log (WAL).
|
||
This could be an issue where prometheus is recovering from an error, or a restart and does not have enough memory to write everything in the WAL.
|
||
We could be running into an issue where we set the request/limits memory lower than the prometheus requires, and the kube scheduler keeps killing prometheus for wanting more memory.</p><p>For this case, we could give it more memory to work to see if it recovers. We should also analyze why the prometheus WAL is getting clogged up.</p><p>In essence, we want to check what has changed so that we suddenly have a high memory spike in our sweet, sweet environment.</p><h2 class="anchor anchorWithStickyNavbar_y2LR" id="the-source">The Source<a class="hash-link" href="#the-source" title="Direct link to heading"></a></h2><p><img alt="Cardinality" src="/assets/images/cardinality-5f722655c50c25a6a91c53884ad19677.webp"></p><p>A lot of prometheus issues revolve around cardinality.
|
||
Memory spikes that break your deployment? Cardinality.
|
||
Prometheus dragging its feet like it's Monday after the log4j (the second one ofc) zero day security breach? Cardinality.
|
||
Not getting that raise since you worked hard the past 16 years without wavering? You bet your ass it's cardinality.
|
||
So, as you can see much of life's problems can be accredited to cardinality.</p><p>In short cardinality of your metrics is the combination of all label values per metric.
|
||
For example, if our metric <code>http_request_total</code> had a label response code, and let's say we support 8 status codes, our cardinality starts off at 8.
|
||
For good measure we want to record the HTTP verb for the request. We support <code>GET POST PUT HEAD</code> which would put the cardinality to 4<!-- -->*<!-- -->8=<strong>32</strong>.
|
||
Now, if someone adds a URL to the metric label (<strong>!!VERY BAD IDEA!!</strong>, but bare with me now) and we have 2 active pages, we'd have a cardinality of 2<!-- -->*<!-- -->4<!-- -->*<!-- -->8=<strong>64</strong>.
|
||
But, imagine someone starts scraping your website for potential vulnerabilities. Imagine all the URLs that will appear, most likely only once.</p><div class="codeBlockContainer_J+bg language-text theme-code-block"><div class="codeBlockContent_csEI text"><pre tabindex="0" class="prism-code language-text codeBlock_rtdJ thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#bfc7d5"><span class="token plain">mywebsite.com/admin.php</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">mywebsite.com/wp/admin.php</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">mywebsite.com/?utm_source=GUID</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">...</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>This would blow up our cardinality to kingdom come. Like you will be out of memory faster than "<a href="https://www.reddit.com/r/ProgrammerHumor/comments/a483yz/those_javascript_devs/" target="_blank" rel="noopener noreferrer">a new super awesome Javascript gamechanger framework</a>" is born.
|
||
Or to quote user <a href="https://www.reddit.com/user/naveen17797/" target="_blank" rel="noopener noreferrer">naveen17797</a> <em>Scientists predict the number of js frameworks may exceed human population by 2020,at that point of time random string generators will be used to name those frameworks.</em></p><p>The point to this story is, be very mindful of how you use labels and cardinality in prometheus, since that will indeed have great impact on your prometheus performance.</p><h2 class="anchor anchorWithStickyNavbar_y2LR" id="the-solution">The Solution<a class="hash-link" href="#the-solution" title="Direct link to heading"></a></h2><p>Since this has never happened to me (never-ever) I found the following solution to be handy.
|
||
Since we can't get prometheus up and running to utilize PromQL to detect the potential issues, we have to find another way to detect high cardinality.
|
||
Therefore, we might want to get our hands dirty with some <code>kubectl exec -it -n monitoring pods/prometheus-prometheus-kube-prometheus-prometheus-0 -- sh</code>, and run the prometheus <code>tsdb</code> analysis too.</p><div class="codeBlockContainer_J+bg language-bash theme-code-block"><div class="codeBlockContent_csEI bash"><pre tabindex="0" class="prism-code language-bash codeBlock_rtdJ thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#bfc7d5"><span class="token plain">/prometheus $ promtool tsdb analyze </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">.</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Which produced the result.</p><div class="codeBlockContainer_J+bg language-text theme-code-block"><div class="codeBlockContent_csEI text"><pre tabindex="0" class="prism-code language-text codeBlock_rtdJ thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Block ID: 01FT8E8YY4THHZ2S7C3G04GJMG</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Duration: 1h59m59.997s</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Series: 564171</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Label names: 285</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings (unique label pairs): 21139</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Postings entries (total label pairs): 6423664</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> Highest cardinality metric names:</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> 11340 haproxy_server_http_responses_total</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">> ...</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>We see the potential issue here, where the <code>haproxy_server_http_responses_total</code> metric is having a super-high cardinality which is growing.
|
||
We need to deal with it, so that our prometheus instance can breathe again. In this particular case, the solution was updating the haproxy.</p><p>... or burn it, up to you.</p><p><img alt="Flame Thrower" src="/assets/images/flame-thrower-56bcf89132356ff53c03ca029d9d0746.webp"></p><h2 class="anchor anchorWithStickyNavbar_y2LR" id="the-further-reading">The Further Reading<a class="hash-link" href="#the-further-reading" title="Direct link to heading"></a></h2><ol><li><a href="https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/wal.md" target="_blank" rel="noopener noreferrer">WAL Definition</a></li><li><a href="https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/" target="_blank" rel="noopener noreferrer">WAL & Checkpoints</a></li><li><a href="https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality" target="_blank" rel="noopener noreferrer">Using TSDB</a></li><li><a href="https://www.robustperception.io/which-are-my-biggest-metrics" target="_blank" rel="noopener noreferrer">Biggest Metrics</a></li><li><a href="https://www.robustperception.io/cardinality-is-key" target="_blank" rel="noopener noreferrer">Cardinality</a></li></ol></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_NBRY padding--none margin-left--sm"><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/prometheus">prometheus</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/cardinality">cardinality</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/devops">devops</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/ops">ops</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/k-8-s">k8s</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/oom">oom</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/memory">memory</a></li></ul></div></footer></article><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="blogPostTitle_d4p0" itemprop="headline"><a itemprop="url" href="/blog/searching-for-search-engines">The never ending search a search engine 2022-01 edition</a></h2><div class="blogPostData_-Im+ margin-vert--md"><time datetime="2022-01-20T00:00:00.000Z" itemprop="datePublished">January 20, 2022</time></div><div class="row margin-top--md margin-bottom--sm"><div class="col col--6 authorCol_8c0z"><div class="avatar margin-bottom--sm"><a href="https://github.com/janhalfar" target="_blank" rel="noopener noreferrer" class="avatar__photo-link avatar__photo"><img class="image_9q7L" src="https://github.com/janhalfar.png" alt="Jan Halfar"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/janhalfar" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Jan Halfar</span></a></div><small class="avatar__subtitle" itemprop="description">foomo maintainer</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><p>While building this website and integrating <a href="https://docsearch.algolia.com" target="_blank" rel="noopener noreferrer">https://docsearch.algolia.com</a> and evaluating another solution by a large company in parallel I could not help to search github and the web for the current state of search engines and search related services.</p><p>Since I had done the same thing about a year ago, I was surprised to see how quickly things are moving atm.</p><h2 class="anchor anchorWithStickyNavbar_y2LR" id="algolia">Algolia<a class="hash-link" href="#algolia" title="Direct link to heading"></a></h2><p>I was blown away by the quality of <a href="https://www.algolia.com" target="_blank" rel="noopener noreferrer">https://www.algolia.com</a> and I wish it was open source, but I guess, we all have to make a living ;)</p><p>To see how awesome a web (search) interface can be check out <a href="https://www.lacoste.com/us/#query=red%20jackets%20for%20men" target="_blank" rel="noopener noreferrer">https://www.lacoste.com/us/#query=red%20jackets%20for%20men</a> </p><p>Apart from that the UI/UX of their backend tools is fantastic.</p><h2 class="anchor anchorWithStickyNavbar_y2LR" id="elastic">Elastic<a class="hash-link" href="#elastic" title="Direct link to heading"></a></h2><p>When it comes to <a href="https://www.elastic.com" target="_blank" rel="noopener noreferrer">https://www.elastic.com</a> I am a bit nervous about the future of the licensing, despite the fact, that I understand their motivation. At the same time the <a href="https://opensearch.org" target="_blank" rel="noopener noreferrer">https://opensearch.org</a> does not seem to be an ampty threat.</p><h2 class="anchor anchorWithStickyNavbar_y2LR" id="typesenseorg">typesense.org<a class="hash-link" href="#typesenseorg" title="Direct link to heading"></a></h2><p>I do not know, who was hiding under a rock, but I had not seen <a href="https://typesense.org" target="_blank" rel="noopener noreferrer">https://typesense.org</a> before and they certainly have a bold claim: <strong><em>"The Open Source Algolia Alternative" / "The Easier To Use ElasticSearch Alternative"</em></strong> </p><p>When looking at <a href="https://github.com/typesense/typesense/graphs/contributors" target="_blank" rel="noopener noreferrer">https://github.com/typesense/typesense/graphs/contributors</a> it seems, that Kishore Nallan has been working on this for a while. Unfourtunately I do not really see a lot of external contributions, C++ does not seem to attract a lot of contribution.</p><h2 class="anchor anchorWithStickyNavbar_y2LR" id="meilisearch">MeiliSearch<a class="hash-link" href="#meilisearch" title="Direct link to heading"></a></h2><p>This Rust project <a href="https://www.meilisearch.com/" target="_blank" rel="noopener noreferrer">https://www.meilisearch.com/</a> seems to be picking up speed and is definetly on the test short list. It is a fresh codebase with siginficant open source contributions and certainly will attract new developers with Rust and a modern architecture.</p><h2 class="anchor anchorWithStickyNavbar_y2LR" id="go-eco-system">Go eco system<a class="hash-link" href="#go-eco-system" title="Direct link to heading"></a></h2><p>Obviously we are very interested in Go powered software and there are a few notable projects. ATM I do not see anything elastic or algolia like, that would be really mature. </p><h3 class="anchor anchorWithStickyNavbar_y2LR" id="bleve--bluge">bleve / bluge<a class="hash-link" href="#bleve--bluge" title="Direct link to heading"></a></h3><p><a href="https://github.com/mschoch" target="_blank" rel="noopener noreferrer">Marty Schoch</a> seems to be the man when it comes down to text indexing libraries in written in Go and bluge seems to be THE library, that is solid and modern, when implementing text search in your Go application.</p><p><a href="https://github.com/blevesearch/bleve" target="_blank" rel="noopener noreferrer">https://github.com/blevesearch/bleve</a>
|
||
<a href="https://github.com/blugelabs/bluge" target="_blank" rel="noopener noreferrer">https://github.com/blugelabs/bluge</a> // next iteration of bleve</p><h4 class="anchor anchorWithStickyNavbar_y2LR" id="projects-using-bluge">projects using bluge<a class="hash-link" href="#projects-using-bluge" title="Direct link to heading"></a></h4><p>All bleeding edge afaik atm - but definitely good places to look at bluge usage</p><p><a href="https://github.com/prabhatsharma/zinc" target="_blank" rel="noopener noreferrer">https://github.com/prabhatsharma/zinc</a>
|
||
<a href="https://github.com/mosuka/phalanx" target="_blank" rel="noopener noreferrer">https://github.com/mosuka/phalanx</a></p><h3 class="anchor anchorWithStickyNavbar_y2LR" id="look-ma-i-made-a-vector-database">Look ma I made a vector database<a class="hash-link" href="#look-ma-i-made-a-vector-database" title="Direct link to heading"></a></h3><p>Gotta take a look at this one - will report later</p><p><a href="https://github.com/semi-technologies/weaviate" target="_blank" rel="noopener noreferrer">https://github.com/semi-technologies/weaviate</a></p></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_NBRY padding--none margin-left--sm"><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/search">search</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/search-engine">search-engine</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/backend">backend</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/go">go</a></li></ul></div></footer></article><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="blogPostTitle_d4p0" itemprop="headline"><a itemprop="url" href="/blog/impact-of-3rd-party-scripts-on-performance">Impact of 3rd party scripts on performance</a></h2><div class="blogPostData_-Im+ margin-vert--md"><time datetime="2022-01-20T00:00:00.000Z" itemprop="datePublished">January 20, 2022</time></div><div class="row margin-top--md margin-bottom--sm"><div class="col col--6 authorCol_8c0z"><div class="avatar margin-bottom--sm"><a href="https://github.com/themre" target="_blank" rel="noopener noreferrer" class="avatar__photo-link avatar__photo"><img class="image_9q7L" src="https://github.com/themre.png" alt="Marko Trebižan"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/themre" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Marko Trebižan</span></a></div><small class="avatar__subtitle" itemprop="description">Frontend Dev</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><h2 class="anchor anchorWithStickyNavbar_y2LR" id="issue-with-performance">Issue with performance<a class="hash-link" href="#issue-with-performance" title="Direct link to heading"></a></h2><p>When building an ecommerce site or an application where performance is a great deal for the users, you need to keep your application fast and responsive. Frontend developers have already many use-cases when the UI becomes laggy and this increases when 3rd party scripts are being included, such as Google Tag Manager or various Live chats (e.g. Intercom).</p><p>This does not only influences the users when using the site but also Lighthouse score gets lower which also influences page rankings. So the most naive and easy way for this is to defer loading of such scripts but when you need to get all the data from the start of the application, such tactic is not an option. So what else can we do?</p><h2 class="anchor anchorWithStickyNavbar_y2LR" id="partytown-to-the-rescue">Partytown to the rescue<a class="hash-link" href="#partytown-to-the-rescue" title="Direct link to heading"></a></h2><p>Developers at BuilderIO created an library <a href="https://github.com/BuilderIO/partytown" target="_blank" rel="noopener noreferrer">Partytown</a> that would allow relocating resources from 3rd party scripts off the main thread.
|
||
We won't dive into specifics how it works, because they explain it nicely on their GitHub page.</p><p>In our stack we use <a href="https://nextjs.org/" target="_blank" rel="noopener noreferrer">Next.js</a> React framework and we will go through the basic steps that will allow us to include Partytown for Google Tag Manager.</p><h3 class="anchor anchorWithStickyNavbar_y2LR" id="setup">Setup<a class="hash-link" href="#setup" title="Direct link to heading"></a></h3><p>Partytown script needs to be located inside our application and live on the same domain. Since we're using monorepo structure, we need to copy this script across all our frontend application. For that we used CopyPlugin webpack plugin in our Next.js config file:</p><div class="codeBlockContainer_J+bg language-javascript theme-code-block"><div class="codeBlockContent_csEI javascript"><pre tabindex="0" class="prism-code language-javascript codeBlock_rtdJ thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#bfc7d5"><span class="token plain">config</span><span class="token punctuation" style="color:rgb(199, 146, 234)">.</span><span class="token property-access">plugins</span><span class="token punctuation" style="color:rgb(199, 146, 234)">.</span><span class="token method function property-access" style="color:rgb(130, 170, 255)">push</span><span class="token punctuation" style="color:rgb(199, 146, 234)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token spread operator" style="color:rgb(137, 221, 255)">...</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token keyword" style="font-style:italic">new</span><span class="token plain"> </span><span class="token class-name" style="color:rgb(255, 203, 107)">CopyPlugin</span><span class="token punctuation" style="color:rgb(199, 146, 234)">(</span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> patterns</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic">// we copy script from node_modules partytown package to `~partytown` folder in our package that serves static files</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token keyword module" style="font-style:italic">from</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> path</span><span class="token punctuation" style="color:rgb(199, 146, 234)">.</span><span class="token method function property-access" style="color:rgb(130, 170, 255)">join</span><span class="token punctuation" style="color:rgb(199, 146, 234)">(</span><span class="token plain">path</span><span class="token punctuation" style="color:rgb(199, 146, 234)">.</span><span class="token method function property-access" style="color:rgb(130, 170, 255)">dirname</span><span class="token punctuation" style="color:rgb(199, 146, 234)">(</span><span class="token plain">require</span><span class="token punctuation" style="color:rgb(199, 146, 234)">.</span><span class="token method function property-access" style="color:rgb(130, 170, 255)">resolve</span><span class="token punctuation" style="color:rgb(199, 146, 234)">(</span><span class="token string" style="color:rgb(195, 232, 141)">'@builder.io/partytown'</span><span class="token punctuation" style="color:rgb(199, 146, 234)">)</span><span class="token punctuation" style="color:rgb(199, 146, 234)">)</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">'lib'</span><span class="token punctuation" style="color:rgb(199, 146, 234)">)</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic">// paths for SSR and client side rendering differ</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> to</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> path</span><span class="token punctuation" style="color:rgb(199, 146, 234)">.</span><span class="token method function property-access" style="color:rgb(130, 170, 255)">join</span><span class="token punctuation" style="color:rgb(199, 146, 234)">(</span><span class="token template-string template-punctuation string" style="color:rgb(195, 232, 141)">`</span><span class="token template-string interpolation interpolation-punctuation punctuation" style="color:rgb(199, 146, 234)">${</span><span class="token template-string interpolation">isServer </span><span class="token template-string interpolation operator" style="color:rgb(137, 221, 255)">?</span><span class="token template-string interpolation"> </span><span class="token template-string interpolation string" style="color:rgb(195, 232, 141)">'..'</span><span class="token template-string interpolation"> </span><span class="token template-string interpolation operator" style="color:rgb(137, 221, 255)">:</span><span class="token template-string interpolation"> </span><span class="token template-string interpolation string" style="color:rgb(195, 232, 141)">'.'</span><span class="token template-string interpolation interpolation-punctuation punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token template-string string" style="color:rgb(195, 232, 141)">/static/assets/</span><span class="token template-string template-punctuation string" style="color:rgb(195, 232, 141)">`</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">'~partytown'</span><span class="token punctuation" style="color:rgb(199, 146, 234)">)</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token punctuation" style="color:rgb(199, 146, 234)">)</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">)</span><span class="token punctuation" style="color:rgb(199, 146, 234)">;</span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Partytown's requirement is that it needs to know what script should it load into own web worker. For that we set script type to <code>text/partytown</code>. This will prevent script to load on initial load.</p><p>Inside <code>_document.tsx</code> we add this:</p><div class="codeBlockContainer_J+bg language-javascript theme-code-block"><div class="codeBlockContent_csEI javascript"><pre tabindex="0" class="prism-code language-javascript codeBlock_rtdJ thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#bfc7d5"><span class="token operator" style="color:rgb(137, 221, 255)"><</span><span class="token maybe-class-name">Head</span><span class="token operator" style="color:rgb(137, 221, 255)">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token spread operator" style="color:rgb(137, 221, 255)">...</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic">// include Partytown and set custom path due to multiple frontends</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token operator" style="color:rgb(137, 221, 255)"><</span><span class="token maybe-class-name">Partytown</span><span class="token plain"> lib</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token template-string template-punctuation string" style="color:rgb(195, 232, 141)">`</span><span class="token template-string interpolation interpolation-punctuation punctuation" style="color:rgb(199, 146, 234)">${</span><span class="token template-string interpolation function" style="color:rgb(130, 170, 255)">addTrailingSlash</span><span class="token template-string interpolation punctuation" style="color:rgb(199, 146, 234)">(</span><span class="token template-string interpolation keyword" style="font-style:italic">this</span><span class="token template-string interpolation punctuation" style="color:rgb(199, 146, 234)">.</span><span class="token template-string interpolation">props</span><span class="token template-string interpolation punctuation" style="color:rgb(199, 146, 234)">.</span><span class="token template-string interpolation">basePath</span><span class="token template-string interpolation punctuation" style="color:rgb(199, 146, 234)">)</span><span class="token template-string interpolation interpolation-punctuation punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token template-string string" style="color:rgb(195, 232, 141)">_next/static/assets/~partytown/</span><span class="token template-string template-punctuation string" style="color:rgb(195, 232, 141)">`</span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token plain"> debug</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token boolean" style="color:rgb(255, 88, 116)">false</span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token plain"> </span><span class="token operator" style="color:rgb(137, 221, 255)">/</span><span class="token operator" style="color:rgb(137, 221, 255)">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic">// tag 3rd party script with partytown type</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token operator" style="color:rgb(137, 221, 255)"><</span><span class="token plain">script type</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token string" style="color:rgb(195, 232, 141)">"text/partytown"</span><span class="token plain"> src</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token template-string template-punctuation string" style="color:rgb(195, 232, 141)">`</span><span class="token template-string string" style="color:rgb(195, 232, 141)">https://www.googletagmanager.com/gtm.js?id=</span><span class="token template-string interpolation interpolation-punctuation punctuation" style="color:rgb(199, 146, 234)">${</span><span class="token template-string interpolation">id</span><span class="token template-string interpolation interpolation-punctuation punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token template-string template-punctuation string" style="color:rgb(195, 232, 141)">`</span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token plain"> </span><span class="token operator" style="color:rgb(137, 221, 255)">/</span><span class="token operator" style="color:rgb(137, 221, 255)">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token spread operator" style="color:rgb(137, 221, 255)">...</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token operator" style="color:rgb(137, 221, 255)"><</span><span class="token operator" style="color:rgb(137, 221, 255)">/</span><span class="token maybe-class-name">Head</span><span class="token operator" style="color:rgb(137, 221, 255)">></span><br></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h2 class="anchor anchorWithStickyNavbar_y2LR" id="results">Results<a class="hash-link" href="#results" title="Direct link to heading"></a></h2><p>So now, does it work? We used one of our large Ecommerce sites to test the landing Lighthouse score. </p><p>This was before adding Partytown:</p><p><img alt="Lighthouse before Partytown" src="/assets/images/lighthouse_before-dc68b4a70a257ec6284efc219a989c7e.jpg"></p><p>Here you can see 2 critical things: almost 1s of total blocking time (TBT) and 9s of time to interactive (TTI). </p><p>After we added Partytown, we got this:</p><p><img alt="Lighthouse after Partytown" src="/assets/images/lighthouse_after-252d381900343cee3782f1d9c3e27442.jpg"></p><p>Time to interactive went from 9s to 6.1s which is almost 33% improvement and total blocking time was reduce by more than 50%! We were more than impressed how easy it was to improve our performances.</p><p>Side note: Both screenshots were compressed using <a href="https://squoosh.app/" target="_blank" rel="noopener noreferrer">Squoosh App</a>.</p><header><h1>Next steps</h1></header><p>After successful testing of Partytown for Google Tag Manager script, we are more than interested in trying it out on our other scripts. One important topic will be to test Partytown with other service-worker related libraries and how to use them together.</p></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_NBRY padding--none margin-left--sm"><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/frontend">frontend</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/performance">performance</a></li></ul></div></footer></article><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="blogPostTitle_d4p0" itemprop="headline"><a itemprop="url" href="/blog/debugging-go-map-races-in-k8s">debugging Go map races in k8s</a></h2><div class="blogPostData_-Im+ margin-vert--md"><time datetime="2022-01-19T00:00:00.000Z" itemprop="datePublished">January 19, 2022</time></div><div class="row margin-top--md margin-bottom--sm"><div class="col col--6 authorCol_8c0z"><div class="avatar margin-bottom--sm"><a href="https://github.com/dreadl0ck" target="_blank" rel="noopener noreferrer" class="avatar__photo-link avatar__photo"><img class="image_9q7L" src="https://github.com/dreadl0ck.png" alt="Philipp Mieden"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/dreadl0ck" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Philipp Mieden</span></a></div><small class="avatar__subtitle" itemprop="description">MSc</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_NBRY padding--none margin-left--sm"><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/go">go</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/debugging">debugging</a></li><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/backend">backend</a></li></ul></div></footer></article><article class="margin-bottom--xl" itemprop="blogPost" itemscope="" itemtype="http://schema.org/BlogPosting"><header><h2 class="blogPostTitle_d4p0" itemprop="headline"><a itemprop="url" href="/blog/welcome-back-2021">Relaunching foomo.org</a></h2><div class="blogPostData_-Im+ margin-vert--md"><time datetime="2021-11-12T00:00:00.000Z" itemprop="datePublished">November 12, 2021</time></div><div class="row margin-top--md margin-bottom--sm"><div class="col col--6 authorCol_8c0z"><div class="avatar margin-bottom--sm"><a href="https://github.com/janhalfar" target="_blank" rel="noopener noreferrer" class="avatar__photo-link avatar__photo"><img class="image_9q7L" src="https://github.com/janhalfar.png" alt="Jan Halfar"></a><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/janhalfar" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Jan Halfar</span></a></div><small class="avatar__subtitle" itemprop="description">foomo maintainer</small></div></div></div></div></header><div class="markdown" itemprop="articleBody"><p>A few years ago we abandoned the previous version of <a href="https://www.foomo.org" target="_blank" rel="noopener noreferrer">https://www.foomo.org</a> as we did not want to maintain the old wordpress installation and the project was only living in README.md in the repos living under <a href="https://www.github.com/foomo" target="_blank" rel="noopener noreferrer">https://www.github.com/foomo</a> .</p><p>As things have grown over time we have decided to re-launch a website / cross project documentation.</p><p>So welcome back and enjoy the view to the past:</p><p><img alt="blast from the past" src="/assets/images/blast-from-the-past-cb642ec62cf61aa0ff51dc863b482b57.png"></p></div><footer class="row docusaurus-mt-lg"><div class="col"><b>Tags:</b><ul class="tags_NBRY padding--none margin-left--sm"><li class="tag_F03v"><a class="tag_WK-t tagRegular_LXbV" href="/blog/tags/foomo">foomo</a></li></ul></div></footer></article><nav class="pagination-nav" aria-label="Blog list page navigation"><div class="pagination-nav__item"></div><div class="pagination-nav__item pagination-nav__item--next"></div></nav></main></div></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">legal</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/etc/imprint">Imprint</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright">© 2022 bestbytes</div></div></div></footer></div>
|
||
<script src="/assets/js/runtime~main.3b4333f9.js"></script>
|
||
<script src="/assets/js/main.681e1e0c.js"></script>
|
||
</body>
|
||
</html> |