Merge pull request #3 from foomo/feature/go-1.24.1

feat: go 1.24.1
This commit is contained in:
Daniel Thomas 2025-03-13 09:23:26 +01:00 committed by GitHub
commit 4a4d334336
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
33 changed files with 786 additions and 181 deletions

133
.github/CODE_OF_CONDUCT.md vendored Normal file
View File

@ -0,0 +1,133 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, caste, color, religion, or sexual
identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall
community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or advances of
any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address,
without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official email address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
info@bestbytes.com.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series of
actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or permanent
ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the
community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.1, available at
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
[https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations

21
.github/CONTRIBUTING.md vendored Normal file
View File

@ -0,0 +1,21 @@
# Contributing
If you want to submit a pull request to fix a bug or enhance an existing
feature, please first open an issue and link to that issue when you
submit your pull request.
If you have any questions about a possible submission, feel free to open
an issue too.
### Pull request process
1. Fork this repository
2. Create a branch in your fork to implement the changes. We recommend using
the issue number as part of your branch name, e.g. `1234-fixes`
3. Ensure that any documentation is updated with the changes that are required
by your fix.
4. Ensure that any samples are updated if the base image has been changed.
5. Submit the pull request. *Do not leave the pull request blank*. Explain exactly
what your changes are meant to do and provide simple steps on how to validate
your changes. Ensure that you reference the issue you created as well.
The pull request will be review before it is merged.

17
.github/ISSUE_TEMPLATE/bug-report.md vendored Normal file
View File

@ -0,0 +1,17 @@
---
name: Bug Report
about: Report a bug you encountered
labels: bug
---
**What happened**:
**What you expected to happen**:
**How to reproduce it (as minimally and precisely as possible)**:
**Anything else we need to know?**:
**Environment**:
- Affected Version:
- OS (e.g: `cat /etc/os-release`):
- Others:

8
.github/ISSUE_TEMPLATE/enhancement.md vendored Normal file
View File

@ -0,0 +1,8 @@
---
name: Enhancement Request
about: Suggest an enhancement
labels: enhancement
---
**What would you like to be added**:
**Why is this needed**:

19
.github/PULL_REQUEST_TEMPLATE.md vendored Normal file
View File

@ -0,0 +1,19 @@
### Type of Change
- [ ] New feature
- [ ] Bug fix
- [ ] Documentation update
- [ ] Refactoring
- [ ] Hotfix
- [ ] Security patch
### Description
_[Provide a detailed explanation of the changes you have made. Include the reasons behind these changes and any relevant context. Link any related issues.]_
### Related Issues
_[If this pull request addresses an issue, please link to it here (e.g., Fixes #123).]_
### Checklist
- [ ] My code adheres to the coding and style guidelines of the project.
- [ ] I have performed a self-review of my own code.
- [ ] I have commented my code, particularly in hard-to-understand areas.
- [ ] I have made corresponding changes to the documentation.

45
.github/SECURITY.md vendored Normal file
View File

@ -0,0 +1,45 @@
# Security Guidelines
## How security is managed on this project
The foomo team and community take security seriously and wants to ensure that
we maintain a secure environment and provide secure solutions for the open
source community. To help us achieve these goals, please note the
following before using this software:
- Review the software license to understand the contributor's obligations in
terms of warranties and suitability for purpose
- For any questions or concerns about security, you can
[create an issue][new-issue] or [report a vulnerability][new-sec-issue]
- We request that you work with our security team and opt for
responsible disclosure using the guidelines below
- All security related issues and pull requests you make should be tagged with
"security" for easy identification
- Please monitor this repository and update your environment in a timely manner
as we release patches and updates
## Responsibly Disclosing Security Bugs
If you find a security bug in this repository, please work with contributors
following responsible disclosure principles and these guidelines:
- Do not submit a normal issue or pull request in our public repository, instead
[report it directly][new-sec-issue].
- We will review your submission and may follow up for additional details
- If you have a patch, we will review it and approve it privately; once approved
for release you can submit it as a pull request publicly in the repository (we
give credit where credit is due)
- We will keep you informed during our investigation, feel free to check in for
a status update
- We will release the fix and publicly disclose the issue as soon as possible,
but want to ensure we due properly due diligence before releasing
- Please do not publicly blog or post about the security issue until after we
have updated the public repo so that other downstream users have an opportunity
to patch
## Contact / Misc
If you have any questions, please reach out directly by [creating an issue][new-issue].
[new-issue]: https://github.com/foomo/typesense/issues/new/choose
[new-sec-issue]: https://github.com/foomo/typesense/security/advisories/new

31
.github/dependabot.yml vendored Normal file
View File

@ -0,0 +1,31 @@
# Please see the documentation for all configuration options:
# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
version: 2
updates:
- package-ecosystem: 'github-actions'
directory: '/'
schedule:
day: 'sunday'
interval: 'weekly'
groups:
github-actions:
patterns: ['*']
- package-ecosystem: 'gomod'
directory: '/'
schedule:
day: 'sunday'
interval: 'weekly'
groups:
gomod-security:
applies-to: security-updates
update-types: ['minor', 'patch']
patterns: ['*']
gomod-update:
applies-to: version-updates
update-types: ['minor', 'patch']
patterns: ['*']
ignore:
- dependency-name: "*"
update-types: ["version-update:semver-major"]

30
.github/workflows/release.yml vendored Normal file
View File

@ -0,0 +1,30 @@
name: Release Tag
on:
push:
tags:
- v*.*.*
workflow_dispatch:
permissions:
contents: write
jobs:
release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-go@v5
with:
check-latest: true
go-version-file: go.mod
- uses: goreleaser/goreleaser-action@v6
with:
version: latest
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

39
.github/workflows/test.yml vendored Normal file
View File

@ -0,0 +1,39 @@
name: Test Branch
on:
push:
branches: [ main ]
pull_request:
merge_group:
workflow_dispatch:
concurrency:
group: "${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}"
cancel-in-progress: true
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
check-latest: true
go-version-file: 'go.mod'
- uses: gotesttools/gotestfmt-action@v2
with:
token: ${{ secrets.GITHUB_TOKEN }}
- uses: golangci/golangci-lint-action@v6
with:
version: latest
args: --timeout=5m
- name: Run tests
run: GO_TEST_TAGS=-skip go test -coverprofile=coverage.out -race -json ./... | gotestfmt
#- uses: coverallsapp/github-action@v2
# with:
# file: coverage.out

40
.gitignore vendored
View File

@ -1,25 +1,25 @@
# If you prefer the allow list template instead of the deny list, see community template:
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
#
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
# Test binary, built with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
.*
*.log
*.out
# Dependency directories (remove the comment below to include it)
# vendor/
/bin/
/tmp/
# Go workspace file
## Git
!.gitignore
## GitHub
!.github/
## Husky
!.husky/
!.husky.yaml
## EditorConfig
!.editorconfig
## Go
!.golangci.yml
!.goreleaser.yml
go.work
go.work.sum
# env file
.env

View File

@ -1,5 +1,10 @@
# yaml-language-server: $schema=https://golangci-lint.run/jsonschema/golangci.jsonschema.json
# https://golangci-lint.run/usage/configuration/
run:
timeout: 5m
go: 1.24.1
build-tags: [safe]
modules-download-mode: readonly
linters-settings:
importas:
@ -7,54 +12,148 @@ linters-settings:
- pkg: '^github.com\/foomo\/typesense\/(.*\/)?([^\/]+)\/?$'
alias: '${2}x' # enforce `x` suffix
no-unaliased: true
# https://golangci-lint.run/usage/linters/#revive
revive:
ignore-generated-header: true
enable-all-rules: true
rules:
- name: unused-parameter
disabled: true
- name: line-length-limit
disabled: true
- name: add-constant
disabled: true
- name: function-result-limit
disabled: true
- name: cognitive-complexity
disabled: true
- name: unused-receiver
disabled: true
- name: use-any
disabled: true
- name: cyclomatic
disabled: true
- name: function-length
disabled: true
- name: max-public-structs
disabled: true
linters:
disable-all: true
enable:
# Enabled by default linters:
- errcheck
- gosimple
- govet
- ineffassign
- staticcheck
## Default linters
- errcheck # errcheck is a program for checking for unchecked errors in Go code. These unchecked errors can be critical bugs in some cases [fast: false, auto-fix: false]
- gosimple # (megacheck) Linter for Go source code that specializes in simplifying code [fast: false, auto-fix: false]
- govet # (vet, vetshadow) Vet examines Go source code and reports suspicious constructs. It is roughly the same as 'go vet' and uses its passes. [fast: false, auto-fix: false]
- ineffassign # Detects when assignments to existing variables are not used [fast: true, auto-fix: false]
- staticcheck # (megacheck) It's a set of rules from staticcheck. It's not the same thing as the staticcheck binary. The author of staticcheck doesn't support or approve the use of staticcheck as a library inside golangci-lint. [fast: false, auto-fix: false]
- unused # (megacheck) Checks Go code for unused constants, variables, functions and types [fast: false, auto-fix: false]
# Disabled by default linters:
- asciicheck # Simple linter to check that your code does not contain non-ASCII identifiers [fast: true, auto-fix: false]
## Recommended linters
- asasalint # check for pass []any as any in variadic func(...any) [fast: false, auto-fix: false]
- asciicheck # checks that all code identifiers does not have non-ASCII symbols in the name [fast: true, auto-fix: false]
- bidichk # Checks for dangerous unicode character sequences [fast: true, auto-fix: false]
# - depguard # Go linter that checks if package imports are in a list of acceptable packages [fast: true, auto-fix: false]
# - dupl # Tool for code clone detection [fast: true, auto-fix: false]
- bodyclose # checks whether HTTP response body is closed successfully [fast: false, auto-fix: false]
- canonicalheader # checks whether net/http.Header uses canonical header [fast: false, auto-fix: false]
- containedctx # containedctx is a linter that detects struct contained context.Context field [fast: false, auto-fix: false]
- contextcheck # check whether the function uses a non-inherited context [fast: false, auto-fix: false]
- copyloopvar # (go >= 1.22) copyloopvar is a linter detects places where loop variables are copied [fast: true, auto-fix: false]
- decorder # check declaration order and count of types, constants, variables and functions [fast: true, auto-fix: false]
- durationcheck # check for two durations multiplied together [fast: false, auto-fix: false]
- errchkjson # Checks types passed to the json encoding functions. Reports unsupported types and reports occations, where the check for the returned error can be omitted. [fast: false, auto-fix: false]
- errname # Checks that sentinel errors are prefixed with the `Err` and error types are suffixed with the `Error`. [fast: false, auto-fix: false]
- errorlint # errorlint is a linter for that can be used to find code that will cause problems with the error wrapping scheme introduced in Go 1.13. [fast: false, auto-fix: false]
- exhaustive # check exhaustiveness of enum switch statements [fast: false, auto-fix: false]
- exptostd # Detects functions from golang.org/x/exp/ that can be replaced by std functions. [auto-fix]
- fatcontext # detects nested contexts in loops and function literals [fast: false, auto-fix: false]
- forbidigo # Forbids identifiers [fast: false, auto-fix: false]
- forcetypeassert # finds forced type assertions [fast: true, auto-fix: false]
- gochecknoinits # Checks that no init functions are present in Go code [fast: true, auto-fix: false]
- gocheckcompilerdirectives # Checks that go compiler directive comments (//go:) are valid. [fast: true, auto-fix: false]
- gochecksumtype # Run exhaustiveness checks on Go "sum types" [fast: false, auto-fix: false]
- goconst # Finds repeated strings that could be replaced by a constant [fast: true, auto-fix: false]
- gocritic # Provides diagnostics that check for bugs, performance and style issues. [fast: false, auto-fix: false]
- goimports # In addition to fixing imports, goimports also formats your code in the same style as gofmt. [fast: true, auto-fix: true]
- gocritic # Provides diagnostics that check for bugs, performance and style issues. [fast: false, auto-fix: true]
- gofmt # Gofmt checks whether code was gofmt-ed. By default this tool runs with -s option to check for code simplification [fast: true, auto-fix: true]
- goheader # Checks is file header matches to pattern [fast: true, auto-fix: true]
- goimports # Check import statements are formatted according to the 'goimport' command. Reformat imports in autofix mode. [fast: true, auto-fix: true]
- gomoddirectives # Manage the use of 'replace', 'retract', and 'excludes' directives in go.mod. [fast: true, auto-fix: false]
- gomodguard # Allow and block list linter for direct Go module dependencies. This is different from depguard where there are different block types for example version constraints and module recommendations. [fast: true, auto-fix: false]
- goprintffuncname # Checks that printf-like functions are named with `f` at the end [fast: true, auto-fix: false]
- gosec # (gas): Inspects source code for security problems [fast: false, auto-fix: false]
- grouper # An analyzer to analyze expression groups. [fast: true, auto-fix: false]
# - importas # Enforces consistent import aliases [fast: false, auto-fix: false]
- maintidx # maintidx measures the maintainability index of each function. [fast: true, auto-fix: false]
- goprintffuncname # Checks that printf-like functions are named with `f` at the end. [fast: true, auto-fix: false]
- gosec # (gas) Inspects source code for security problems [fast: false, auto-fix: false]
- gosmopolitan # Report certain i18n/l10n anti-patterns in your Go codebase [fast: false, auto-fix: false]
- grouper # Analyze expression groups. [fast: true, auto-fix: false]
- iface # Detect the incorrect use of interfaces, helping developers avoid interface pollution. [fast: false, auto-fix: false]
- importas # Enforces consistent import aliases [fast: false, auto-fix: false]
- inamedparam # reports interfaces with unnamed method parameters [fast: true, auto-fix: false]
- intrange # (go >= 1.22) intrange is a linter to find places where for loops could make use of an integer range. [fast: true, auto-fix: false]
- loggercheck # (logrlint) Checks key value pairs for common logger libraries (kitlog,klog,logr,zap). [fast: false, auto-fix: false]
- makezero # Finds slice declarations with non-zero initial length [fast: false, auto-fix: false]
- misspell # Finds commonly misspelled English words in comments [fast: true, auto-fix: true]
- nakedret # Finds naked returns in functions greater than a specified function length [fast: true, auto-fix: false]
- nestif # Reports deeply nested if statements [fast: true, auto-fix: false]
- mirror # reports wrong mirror patterns of bytes/strings usage [fast: false, auto-fix: true]
- misspell # Finds commonly misspelled English words [fast: true, auto-fix: true]
- musttag # enforce field tags in (un)marshaled structs [fast: false, auto-fix: false]
- nakedret # Checks that functions with naked returns are not longer than a maximum size (can be zero). [fast: true, auto-fix: false]
- nilerr # Finds the code that returns nil even if it checks that the error is not nil. [fast: false, auto-fix: false]
- nilnesserr # Reports constructs that checks for err != nil, but returns a different nil value error.
- nilnil # Checks that there is no simultaneous return of `nil` error and an invalid value. [fast: false, auto-fix: false]
- noctx # noctx finds sending http request without context.Context [fast: false, auto-fix: false]
- nolintlint # Reports ill-formed or insufficient nolint directives [fast: true, auto-fix: false]
# - nonamedreturns # Reports all named returns [fast: false, auto-fix: false]
- noctx # Finds sending http request without context.Context [fast: false, auto-fix: false]
- nolintlint # Reports ill-formed or insufficient nolint directives [fast: true, auto-fix: true]
- nonamedreturns # Reports all named returns [fast: false, auto-fix: false]
- nosprintfhostport # Checks for misuse of Sprintf to construct a host with port in a URL. [fast: true, auto-fix: false]
- prealloc # Finds slice declarations that could potentially be pre-allocated [fast: true, auto-fix: false]
- paralleltest # Detects missing usage of t.Parallel() method in your Go test [fast: false, auto-fix: false]
- predeclared # find code that shadows one of Go's predeclared identifiers [fast: true, auto-fix: false]
- promlinter # Check Prometheus metrics naming via promlint [fast: true, auto-fix: false]
- reassign # Checks that package variables are not reassigned [fast: false, auto-fix: false]
- recvcheck # checks for receiver type consistency [fast: false, auto-fix: false]
- revive # Fast, configurable, extensible, flexible, and beautiful linter for Go. Drop-in replacement of golint. [fast: false, auto-fix: false]
- tagliatelle # Checks the struct tags. [fast: true, auto-fix: false]
- rowserrcheck # checks whether Rows.Err of rows is checked successfully [fast: false, auto-fix: false]
- spancheck # Checks for mistakes with OpenTelemetry/Census spans. [fast: false, auto-fix: false]
- sqlclosecheck # Checks that sql.Rows, sql.Stmt, sqlx.NamedStmt, pgx.Query are closed. [fast: false, auto-fix: false]
- stylecheck # Stylecheck is a replacement for golint [fast: false, auto-fix: false]
- testableexamples # linter checks if examples are testable (have an expected output) [fast: true, auto-fix: false]
- testifylint # Checks usage of github.com/stretchr/testify. [fast: false, auto-fix: false]
- testpackage # linter that makes you use a separate _test package [fast: true, auto-fix: false]
- thelper # thelper detects golang test helpers without t.Helper() call and checks the consistency of test helpers [fast: false, auto-fix: false]
- thelper # thelper detects tests helpers which is not start with t.Helper() method. [fast: false, auto-fix: false]
- tparallel # tparallel detects inappropriate usage of t.Parallel() method in your Go test codes. [fast: false, auto-fix: false]
- unconvert # Remove unnecessary type conversions [fast: false, auto-fix: false]
- unparam # Reports unused function parameters [fast: false, auto-fix: false]
- usestdlibvars # A linter that detect the possibility to use variables/constants from the Go standard library. [fast: true, auto-fix: false]
- wastedassign # wastedassign finds wasted assignment statements. [fast: false, auto-fix: false]
- whitespace # Tool for detection of leading and trailing whitespace [fast: true, auto-fix: true]
disable:
- unused
- usetesting # Reports uses of functions with replacement inside the testing package. [auto-fix]
- wastedassign # Finds wasted assignment statements [fast: false, auto-fix: false]
- whitespace # Whitespace is a linter that checks for unnecessary newlines at the start and end of functions, if, for, etc. [fast: true, auto-fix: true]
## Discouraged linters
#- cyclop # checks function and package cyclomatic complexity [fast: false, auto-fix: false]
#- depguard # Go linter that checks if package imports are in a list of acceptable packages [fast: true, auto-fix: false]
#- dogsled # Checks assignments with too many blank identifiers (e.g. x, _, _, _, := f()) [fast: true, auto-fix: false]
#- dupl # Tool for code clone detection [fast: true, auto-fix: false]
#- dupword # checks for duplicate words in the source code [fast: true, auto-fix: false]
#- dogsled # Checks assignments with too many blank identifiers (e.g. x, _, _, _, := f()) [fast: true, auto-fix: false]
#- err113 # Go linter to check the errors handling expressions [fast: false, auto-fix: false]
#- exhaustruct # Checks if all structure fields are initialized [fast: false, auto-fix: false]
#- funlen # Tool for detection of long functions [fast: true, auto-fix: false]
#- gci # Gci controls Go package import order and makes it always deterministic. [fast: true, auto-fix: true]
#- ginkgolinter # enforces standards of using ginkgo and gomega [fast: false, auto-fix: false]
#- gochecknoglobals # Check that no global variables exist. [fast: false, auto-fix: false]
#- gochecknoinits # Checks that no init functions are present in Go code [fast: true, auto-fix: false]
#- gocognit # Computes and checks the cognitive complexity of functions [fast: true, auto-fix: false]
#- gocyclo # Computes and checks the cyclomatic complexity of functions [fast: true, auto-fix: false]
#- godot # Check if comments end in a period [fast: true, auto-fix: true]
#- godox # Tool for detection of comment keywords [fast: true, auto-fix: false]
#- gofumpt # Gofumpt checks whether code was gofumpt-ed. [fast: true, auto-fix: true]
#- interfacebloat # A linter that checks the number of methods inside an interface. [fast: true, auto-fix: false]
#- intrange # (go >= 1.22) intrange is a linter to find places where for loops could make use of an integer range. [fast: true, auto-fix: false]
#- ireturn # Accept Interfaces, Return Concrete Types [fast: false, auto-fix: false]
#- lll # Reports long lines [fast: true, auto-fix: false]
#- maintidx # maintidx measures the maintainability index of each function. [fast: true, auto-fix: false]
#- nestif # Reports deeply nested if statements [fast: true, auto-fix: false]
#- nlreturn # nlreturn checks for a new line before return and branch statements to increase code clarity [fast: true, auto-fix: false]
#- mnd # An analyzer to detect magic numbers. [fast: true, auto-fix: false]
#- perfsprint # Checks that fmt.Sprintf can be replaced with a faster alternative. [fast: false, auto-fix: false]
#- prealloc # Finds slice declarations that could potentially be pre-allocated [fast: true, auto-fix: false]
#- protogetter # Reports direct reads from proto message fields when getters should be used [fast: false, auto-fix: true]
#- sloglint # ensure consistent code style when using log/slog [fast: false, auto-fix: false]
#- tagalign # check that struct tags are well aligned [fast: true, auto-fix: true]
#- tagliatelle # Checks the struct tags. [fast: true, auto-fix: false]
#- unparam # Reports unused function parameters [fast: false, auto-fix: false]
#- varnamelen # checks that the length of a variable's name matches its scope [fast: false, auto-fix: false]
#- wrapcheck # Checks that errors returned from external packages are wrapped [fast: false, auto-fix: false]
#- wsl # add or remove empty lines [fast: true, auto-fix: false]
#- zerologlint # Detects the wrong usage of `zerolog` that a user forgets to dispatch with `Send` or `Msg` [fast: false, auto-fix: false]

View File

@ -1,34 +1,7 @@
version: 2
builds:
- skip: true
changelog:
filters:
exclude:
- "^wip"
- "^test"
- "^docs"
- "^chore"
- "^style"
- "go mod tidy"
- "merge conflict"
- "Merge pull request"
- "Merge remote-tracking branch"
- "Merge branch"
groups:
- title: Features
regexp: '^.*?feat(\([[:word:]]+\))??!?:.+$'
order: 0
- title: Dependency updates
regexp: '^.*?(feat|fix)\(deps\)!?:.+$'
order: 100
- title: "Bug fixes"
regexp: '^.*?fix(\([[:word:]]+\))??!?:.+$'
order: 150
- title: "Security"
regexp: '^.*?sec(\([[:word:]]+\))??!?:.+$'
order: 200
- title: "Performace"
regexp: '^.*?perf(\([[:word:]]+\))??!?:.+$'
order: 250
- title: Other
order: 999
use: github-native

15
.husky.yaml Normal file
View File

@ -0,0 +1,15 @@
hooks:
pre-commit:
- golangci-lint run --fast
- husky lint-staged
commit-msg:
# only execute if not in a merge
- if [[ -z $(git rev-parse -q --verify MERGE_HEAD) ]]; then husky lint-commit; fi
lint-staged:
'*.go':
- goimports -l -w
lint-commit:
types: '^(feat|fix|build|chore|docs|perf|refactor|revert|style|test|wip)$'
header: '^(?P<type>\w+)(\((?P<scope>[\w/.-]+)\))?(?P<breaking>!)?:( +)?(?P<header>.+)'

3
.husky/applypatch-msg Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/commit-msg Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/fsmonitor-watchman Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/post-update Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/pre-applypatch Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/pre-commit Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/pre-merge-commit Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/pre-push Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/pre-rebase Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/pre-receive Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/prepare-commit-msg Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/push-to-checkout Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/sendemail-validate Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

3
.husky/update Executable file
View File

@ -0,0 +1,3 @@
#!/bin/sh
husky hook $(basename "$0") $*

129
README.md
View File

@ -0,0 +1,129 @@
# Typesense API
[![Build Status](https://github.com/foomo/typesense/actions/workflows/test.yml/badge.svg?branch=main&event=push)](https://github.com/foomo/typesense/actions/workflows/test.yml)
[![Go Report Card](https://goreportcard.com/badge/github.com/foomo/typesense)](https://goreportcard.com/report/github.com/foomo/typesense)
[![GoDoc](https://godoc.org/github.com/foomo/typesense?status.svg)](https://godoc.org/github.com/foomo/typesense)
## Overview
This package provides an API for managing and searching Typesense collections. It offers functionalities for initializing, indexing, searching, and maintaining Typesense collections and aliases.
## Features
- **Initialization**: Ensures that all aliases point to the latest revision-based collections.
- **Health Check**: Verifies if the Typesense client is operational.
- **Index Management**: Lists, creates, and updates index collections.
- **Document Upsertion**: Bulk upsert support for indexing documents.
- **Search Operations**: Provides simple and advanced search capabilities.
- **Revision Management**: Supports committing and reverting indexing revisions.
## Installation
To use this package, add it as a dependency in your Go project:
```sh
go get github.com/foomo/typesense
```
## Usage example
```go
import (
"context"
"time"
"github.com/foomo/contentserver/client"
"github.com/foomo/keel/config"
"github.com/foomo/keel/log"
typesenseapi "github.com/foomo/typesense/pkg/api"
typesenseindexing "github.com/foomo/typesense/pkg/indexing"
"github.com/typesense/typesense-go/v3/typesense"
)
func main() {
ctx := context.Background()
l := log.Logger()
// contentserver client
csClient, errClient := client.NewHTTPClient("contentserver_url")
log.Must(l, errClient, "could not get contentserver client")
// contentful client
// provide list of content types
cfClients := contentful.NewDefaultContentfulClients(ctx, l, contentful_types, true)
cfClients.UpdateCache()
cfClients.Client.ClientStats()
// create typesense client
typesenseClient := typesense.NewClient(
typesense.WithConnectionTimeout(2*time.Minute),
typesense.WithServer("typesense_server"),
typesense.WithAPIKey("typesense_api_key"),
)
// configure document provider
documentProvider := typesenseindexing.NewContentServer(
l, csClient,
GetDocumentProviderFunctions(...), // retrieve document provider functions
supportedMimeTypes, // provide supported mime types
)
// create typesense api
// create indexDocument and returnType
api := typesenseapi.NewBaseAPI[indexDocument, returnType](
l,
typesenseClient,
collectionSchemas, //map[IndexID]*api.CollectionSchema
presetUpsertSchema, //*api.PresetUpsertSchema
)
// create typesense indexer
indexer := typesenseindexing.NewBaseIndexer(
l,
api,
documentProvider,
)
// run indexer
err := indexer.Run(ctx)
log.Must(l, err, "could not run indexer")
}
```
### Health Check
```go
err := apiInstance.Healthz(context.Background())
if err != nil {
log.Fatalf("Health check failed: %v", err)
}
```
### Searching Documents
#### Simple Search
```go
results, scores, total, err := apiInstance.SimpleSearch(context.Background(), "products", "laptop", nil, 1, 10, "price:desc")
if err != nil {
log.Fatalf("Search failed: %v", err)
}
log.Printf("Found %d results", total)
```
#### Advanced Search
```go
searchParams := &api.SearchCollectionParams{
Q: pointer.String("laptop"),
SortBy: pointer.String("price:desc"),
}
results, scores, total, err = apiInstance.ExpertSearch(context.Background(), "products", searchParams)
if err != nil {
log.Fatalf("Advanced search failed: %v", err)
}
log.Printf("Found %d results", total)
```
## How to Contribute
Please refer to the [CONTRIBUTING](.github/CONTRIBUTING.md) details and follow the [CODE_OF_CONDUCT](.github/CODE_OF_CONDUCT.md) and [SECURITY](.github/SECURITY.md) guidelines.
## License
Distributed under MIT License, please see license file within the code for more details.
_Made with ♥ [foomo](https://www.foomo.org) by [bestbytes](https://www.bestbytes.com)_

2
go.mod
View File

@ -1,6 +1,6 @@
module github.com/foomo/typesense
go 1.24.0
go 1.24.1
require (
github.com/foomo/contentserver v1.11.2

View File

@ -7,7 +7,7 @@ import (
"strconv"
"time"
pkgtypesense "github.com/foomo/typesense/pkg"
pkgx "github.com/foomo/typesense/pkg"
"github.com/typesense/typesense-go/v3/typesense"
"github.com/typesense/typesense-go/v3/typesense/api"
"github.com/typesense/typesense-go/v3/typesense/api/pointer"
@ -19,16 +19,16 @@ const defaultSearchPresetName = "default"
type BaseAPI[indexDocument any, returnType any] struct {
l *zap.Logger
client *typesense.Client
collections map[pkgtypesense.IndexID]*api.CollectionSchema
collections map[pkgx.IndexID]*api.CollectionSchema
preset *api.PresetUpsertSchema
revisionID pkgtypesense.RevisionID
revisionID pkgx.RevisionID
}
func NewBaseAPI[indexDocument any, returnType any](
l *zap.Logger,
client *typesense.Client,
collections map[pkgtypesense.IndexID]*api.CollectionSchema,
collections map[pkgx.IndexID]*api.CollectionSchema,
preset *api.PresetUpsertSchema,
) *BaseAPI[indexDocument, returnType] {
return &BaseAPI[indexDocument, returnType]{
@ -48,11 +48,11 @@ func (b *BaseAPI[indexDocument, returnType]) Healthz(_ context.Context) error {
}
// Indices returns a list of all configured index IDs
func (b *BaseAPI[indexDocument, returnType]) Indices() ([]pkgtypesense.IndexID, error) {
func (b *BaseAPI[indexDocument, returnType]) Indices() ([]pkgx.IndexID, error) {
if len(b.collections) == 0 {
return nil, errors.New("no collections configured")
}
indices := make([]pkgtypesense.IndexID, 0, len(b.collections))
indices := make([]pkgx.IndexID, 0, len(b.collections))
for index := range b.collections {
indices = append(indices, index)
}
@ -84,19 +84,19 @@ func (b *BaseAPI[indexDocument, returnType]) Indices() ([]pkgtypesense.IndexID,
// The system is considered valid if there is one alias for each collection and the collections
// are correctly linked to their respective aliases.
// The function sets the revisionID that is currently linked to the aliases internally.
func (b *BaseAPI[indexDocument, returnType]) Initialize(ctx context.Context) (pkgtypesense.RevisionID, error) {
b.l.Info("Initializing Typesense collections and aliases...")
func (b *BaseAPI[indexDocument, returnType]) Initialize(ctx context.Context) (pkgx.RevisionID, error) {
b.l.Info("initializing typesense collections and aliases...")
// Step 1: Check Typesense connection
if _, err := b.client.Health(ctx, 5*time.Second); err != nil {
b.l.Error("Typesense health check failed", zap.Error(err))
b.l.Error("typesense health check failed", zap.Error(err))
return "", err
}
// Step 2: Retrieve existing aliases and collections
aliases, err := b.client.Aliases().Retrieve(ctx)
if err != nil {
b.l.Error("Failed to retrieve aliases", zap.Error(err))
b.l.Error("failed to retrieve aliases", zap.Error(err))
return "", err
}
@ -106,12 +106,12 @@ func (b *BaseAPI[indexDocument, returnType]) Initialize(ctx context.Context) (pk
}
// Step 3: Track latest revisions per alias
latestRevisions := make(map[pkgtypesense.IndexID]pkgtypesense.RevisionID)
aliasMappings := make(map[pkgtypesense.IndexID]string) // Tracks alias-to-collection mappings
latestRevisions := make(map[pkgx.IndexID]pkgx.RevisionID)
aliasMappings := make(map[pkgx.IndexID]string) // Tracks alias-to-collection mappings
for _, alias := range aliases {
collectionName := alias.CollectionName
indexID := pkgtypesense.IndexID(*alias.Name)
indexID := pkgx.IndexID(*alias.Name)
revisionID := extractRevisionID(collectionName, string(indexID))
// Ensure alias points to an existing collection
@ -119,18 +119,18 @@ func (b *BaseAPI[indexDocument, returnType]) Initialize(ctx context.Context) (pk
latestRevisions[indexID] = revisionID
aliasMappings[indexID] = collectionName
} else {
b.l.Warn("Alias points to missing collection, resetting", zap.String("alias", string(indexID)))
b.l.Warn("alias points to missing collection, resetting", zap.String("alias", string(indexID)))
}
}
// Step 4: Ensure all aliases are correctly mapped to collections and create a new revision
newRevisionID := b.generateRevisionID()
b.l.Info("Generated new revision", zap.String("revisionID", string(newRevisionID)))
b.l.Info("generated new revision", zap.String("revisionID", string(newRevisionID)))
for indexID, schema := range b.collections {
collectionName := formatCollectionName(indexID, newRevisionID)
b.l.Warn("Creating new collection & alias",
b.l.Warn("creating new collection & alias",
zap.String("index", string(indexID)),
zap.String("new_collection", collectionName),
)
@ -153,24 +153,24 @@ func (b *BaseAPI[indexDocument, returnType]) Initialize(ctx context.Context) (pk
if b.preset != nil {
_, err := b.client.Presets().Upsert(ctx, defaultSearchPresetName, b.preset)
if err != nil {
b.l.Error("Failed to upsert search preset", zap.Error(err))
b.l.Error("failed to upsert search preset", zap.Error(err))
return "", err
}
}
b.l.Info("Initialization completed", zap.String("revisionID", string(b.revisionID)))
b.l.Info("initialization completed", zap.String("revisionID", string(b.revisionID)))
return b.revisionID, nil
}
func (b *BaseAPI[indexDocument, returnType]) UpsertDocuments(
ctx context.Context,
revisionID pkgtypesense.RevisionID,
indexID pkgtypesense.IndexID,
revisionID pkgx.RevisionID,
indexID pkgx.IndexID,
documents []*indexDocument,
) error {
if len(documents) == 0 {
b.l.Warn("No documents provided for upsert", zap.String("index", string(indexID)))
b.l.Warn("no documents provided for upsert", zap.String("index", string(indexID)))
return nil
}
@ -190,7 +190,7 @@ func (b *BaseAPI[indexDocument, returnType]) UpsertDocuments(
importResults, err := b.client.Collection(collectionName).Documents().Import(ctx, docInterfaces, params)
if err != nil {
b.l.Error("Failed to bulk upsert documents", zap.String("collection", collectionName), zap.Error(err))
b.l.Error("failed to bulk upsert documents", zap.String("collection", collectionName), zap.Error(err))
return err
}
@ -201,14 +201,14 @@ func (b *BaseAPI[indexDocument, returnType]) UpsertDocuments(
successCount++
} else {
failureCount++
b.l.Warn("Document failed to upsert",
b.l.Warn("document failed to upsert",
zap.String("collection", collectionName),
zap.String("error", result.Error),
)
}
}
b.l.Info("Bulk upsert completed",
b.l.Info("bulk upsert completed",
zap.String("collection", collectionName),
zap.Int("successful_documents", successCount),
zap.Int("failed_documents", failureCount),
@ -220,7 +220,7 @@ func (b *BaseAPI[indexDocument, returnType]) UpsertDocuments(
// it will update the aliases to point to the new revision
// additionally it will remove all old collections that are not linked to an alias
// keeping only the latest revision and the one before
func (b *BaseAPI[indexDocument, returnType]) CommitRevision(ctx context.Context, revisionID pkgtypesense.RevisionID) error {
func (b *BaseAPI[indexDocument, returnType]) CommitRevision(ctx context.Context, revisionID pkgx.RevisionID) error {
for indexID := range b.collections {
alias := string(indexID)
newCollectionName := formatCollectionName(indexID, revisionID)
@ -231,15 +231,15 @@ func (b *BaseAPI[indexDocument, returnType]) CommitRevision(ctx context.Context,
CollectionName: newCollectionName,
})
if err != nil {
b.l.Error("Failed to update alias", zap.String("alias", alias), zap.Error(err))
b.l.Error("failed to update alias", zap.String("alias", alias), zap.Error(err))
return err
}
b.l.Info("Updated alias", zap.String("alias", alias), zap.String("collection", newCollectionName))
b.l.Info("updated alias", zap.String("alias", alias), zap.String("collection", newCollectionName))
// Step 2: Clean up old collections (keep only the last two)
err = b.pruneOldCollections(ctx, alias, newCollectionName)
if err != nil {
b.l.Error("Failed to clean up old collections", zap.String("alias", alias), zap.Error(err))
b.l.Error("failed to clean up old collections", zap.String("alias", alias), zap.Error(err))
}
}
@ -247,18 +247,18 @@ func (b *BaseAPI[indexDocument, returnType]) CommitRevision(ctx context.Context,
}
// RevertRevision will remove the collections created for the given revisionID
func (b *BaseAPI[indexDocument, returnType]) RevertRevision(ctx context.Context, revisionID pkgtypesense.RevisionID) error {
func (b *BaseAPI[indexDocument, returnType]) RevertRevision(ctx context.Context, revisionID pkgx.RevisionID) error {
for indexID := range b.collections {
collectionName := formatCollectionName(indexID, revisionID)
// Step 1: Delete the collection safely
_, err := b.client.Collection(collectionName).Delete(ctx)
if err != nil {
b.l.Error("Failed to delete collection", zap.String("collection", collectionName), zap.Error(err))
b.l.Error("failed to delete collection", zap.String("collection", collectionName), zap.Error(err))
return err
}
b.l.Info("Reverted and deleted collection", zap.String("collection", collectionName))
b.l.Info("reverted and deleted collection", zap.String("collection", collectionName))
}
return nil
@ -268,12 +268,12 @@ func (b *BaseAPI[indexDocument, returnType]) RevertRevision(ctx context.Context,
// it will return the documents and the scores
func (b *BaseAPI[indexDocument, returnType]) SimpleSearch(
ctx context.Context,
index pkgtypesense.IndexID,
index pkgx.IndexID,
q string,
filterBy map[string][]string,
page, perPage int,
sortBy string,
) ([]returnType, pkgtypesense.Scores, int, error) {
) ([]returnType, pkgx.Scores, int, error) {
// Call buildSearchParams but also set QueryBy explicitly
parameters := buildSearchParams(q, filterBy, page, perPage, sortBy)
parameters.QueryBy = pointer.String("title")
@ -285,18 +285,18 @@ func (b *BaseAPI[indexDocument, returnType]) SimpleSearch(
// it will return the documents, scores, and totalResults
func (b *BaseAPI[indexDocument, returnType]) ExpertSearch(
ctx context.Context,
indexID pkgtypesense.IndexID,
indexID pkgx.IndexID,
parameters *api.SearchCollectionParams,
) ([]returnType, pkgtypesense.Scores, int, error) {
) ([]returnType, pkgx.Scores, int, error) {
if parameters == nil {
b.l.Error("Search parameters are nil")
b.l.Error("search parameters are nil")
return nil, nil, 0, errors.New("search parameters cannot be nil")
}
collectionName := string(indexID) // digital-bks-at-de
searchResponse, err := b.client.Collection(collectionName).Documents().Search(ctx, parameters)
if err != nil {
b.l.Error("Failed to perform search", zap.String("index", collectionName), zap.Error(err))
b.l.Error("failed to perform search", zap.String("index", collectionName), zap.Error(err))
return nil, nil, 0, err
}
// Extract totalResults from the search response
@ -304,16 +304,16 @@ func (b *BaseAPI[indexDocument, returnType]) ExpertSearch(
// Ensure Hits is not empty before proceeding
if searchResponse.Hits == nil || len(*searchResponse.Hits) == 0 {
b.l.Warn("Search response contains no hits", zap.String("index", collectionName))
b.l.Warn("search response contains no hits", zap.String("index", collectionName))
return nil, nil, totalResults, nil
}
results := make([]returnType, len(*searchResponse.Hits))
scores := make(pkgtypesense.Scores)
scores := make(pkgx.Scores)
for i, hit := range *searchResponse.Hits {
if hit.Document == nil {
b.l.Warn("Hit document is nil", zap.String("index", collectionName))
b.l.Warn("hit document is nil", zap.String("index", collectionName))
continue
}
@ -322,15 +322,19 @@ func (b *BaseAPI[indexDocument, returnType]) ExpertSearch(
// Extract document ID safely
docID, ok := docMap["id"].(string)
if !ok {
b.l.Warn("Missing or invalid document ID in search result")
b.l.Warn("missing or invalid document ID in search result")
continue
}
// Convert hit to JSON and then unmarshal into returnType
hitJSON, _ := json.Marshal(docMap)
hitJSON, err := json.Marshal(docMap)
if err != nil {
b.l.Warn("failed to marshal document to JSON", zap.String("index", collectionName), zap.Error(err))
continue
}
var doc returnType
if err := json.Unmarshal(hitJSON, &doc); err != nil {
b.l.Warn("Failed to unmarshal search result", zap.String("index", collectionName), zap.Error(err))
b.l.Warn("failed to unmarshal JSON into returnType", zap.String("index", collectionName), zap.Error(err))
continue
}
@ -340,17 +344,17 @@ func (b *BaseAPI[indexDocument, returnType]) ExpertSearch(
if score, err := strconv.Atoi(*hit.TextMatchInfo.Score); err == nil {
index = score
} else {
b.l.Warn("Invalid score value", zap.String("score", *hit.TextMatchInfo.Score), zap.Error(err))
b.l.Warn("invalid score value", zap.String("score", *hit.TextMatchInfo.Score), zap.Error(err))
}
}
scores[pkgtypesense.DocumentID(docID)] = pkgtypesense.Score{
ID: pkgtypesense.DocumentID(docID),
scores[pkgx.DocumentID(docID)] = pkgx.Score{
ID: pkgx.DocumentID(docID),
Index: index,
}
}
b.l.Info("Search completed",
b.l.Info("search completed",
zap.String("index", collectionName),
zap.Int("results_count", len(results)),
zap.Int("total_results", totalResults),

View File

@ -7,7 +7,7 @@ import (
"strings"
"time"
pkgtypesense "github.com/foomo/typesense/pkg"
pkgx "github.com/foomo/typesense/pkg"
"github.com/typesense/typesense-go/v3/typesense/api"
"github.com/typesense/typesense-go/v3/typesense/api/pointer"
"go.uber.org/zap"
@ -59,15 +59,15 @@ func formatFilterQuery(filterBy map[string][]string) string {
return strings.Join(filterClauses, " && ")
}
func (b *BaseAPI[indexDocument, returnType]) generateRevisionID() pkgtypesense.RevisionID {
return pkgtypesense.RevisionID(time.Now().Format("2006-01-02-15-04")) // "YYYY-MM-DD-HH-MM"
func (b *BaseAPI[indexDocument, returnType]) generateRevisionID() pkgx.RevisionID {
return pkgx.RevisionID(time.Now().Format("2006-01-02-15-04")) // "YYYY-MM-DD-HH-MM"
}
func formatCollectionName(indexID pkgtypesense.IndexID, revisionID pkgtypesense.RevisionID) string {
func formatCollectionName(indexID pkgx.IndexID, revisionID pkgx.RevisionID) string {
return fmt.Sprintf("%s-%s", indexID, revisionID)
}
func extractRevisionID(collectionName, name string) pkgtypesense.RevisionID {
func extractRevisionID(collectionName, name string) pkgx.RevisionID {
if !strings.HasPrefix(collectionName, name+"-") {
return ""
}
@ -79,16 +79,16 @@ func extractRevisionID(collectionName, name string) pkgtypesense.RevisionID {
return ""
}
return pkgtypesense.RevisionID(revisionID)
return pkgx.RevisionID(revisionID)
}
// ensureAliasMapping ensures an alias correctly points to the specified collection.
func (b *BaseAPI[indexDocument, returnType]) ensureAliasMapping(ctx context.Context, indexID pkgtypesense.IndexID, collectionName string) error {
func (b *BaseAPI[indexDocument, returnType]) ensureAliasMapping(ctx context.Context, indexID pkgx.IndexID, collectionName string) error {
_, err := b.client.Aliases().Upsert(ctx, string(indexID), &api.CollectionAliasSchema{
CollectionName: collectionName,
})
if err != nil {
b.l.Error("Failed to upsert alias",
b.l.Error("failed to upsert alias",
zap.String("alias", string(indexID)),
zap.String("collection", collectionName),
zap.Error(err),
@ -101,7 +101,7 @@ func (b *BaseAPI[indexDocument, returnType]) pruneOldCollections(ctx context.Con
// Step 1: Retrieve all collections
collections, err := b.client.Collections().Retrieve(ctx)
if err != nil {
b.l.Error("Failed to retrieve collections", zap.Error(err))
b.l.Error("failed to retrieve collections", zap.Error(err))
return err
}
@ -123,9 +123,9 @@ func (b *BaseAPI[indexDocument, returnType]) pruneOldCollections(ctx context.Con
for _, col := range toDelete {
_, err := b.client.Collection(col).Delete(ctx)
if err != nil {
b.l.Error("Failed to delete collection", zap.String("collection", col), zap.Error(err))
b.l.Error("failed to delete collection", zap.String("collection", col), zap.Error(err))
} else {
b.l.Info("Deleted old collection", zap.String("collection", col))
b.l.Info("deleted old collection", zap.String("collection", col))
}
}
}
@ -137,7 +137,7 @@ func (b *BaseAPI[indexDocument, returnType]) pruneOldCollections(ctx context.Con
func (b *BaseAPI[indexDocument, returnType]) fetchExistingCollections(ctx context.Context) (map[string]bool, error) {
collections, err := b.client.Collections().Retrieve(ctx)
if err != nil {
b.l.Error("Failed to retrieve collections", zap.Error(err))
b.l.Error("failed to retrieve collections", zap.Error(err))
return nil, err
}
@ -158,7 +158,7 @@ func (b *BaseAPI[indexDocument, returnType]) createCollectionIfNotExists(ctx con
}
if existingCollections[collectionName] {
b.l.Info("Collection already exists, skipping creation", zap.String("collection", collectionName))
b.l.Info("collection already exists, skipping creation", zap.String("collection", collectionName))
return nil
}
@ -166,10 +166,10 @@ func (b *BaseAPI[indexDocument, returnType]) createCollectionIfNotExists(ctx con
schema.Name = collectionName
_, err = b.client.Collections().Create(ctx, schema)
if err != nil {
b.l.Error("Failed to create collection", zap.String("collection", collectionName), zap.Error(err))
b.l.Error("failed to create collection", zap.String("collection", collectionName), zap.Error(err))
return err
}
b.l.Info("Created new collection", zap.String("collection", collectionName))
b.l.Info("created new collection", zap.String("collection", collectionName))
return nil
}

View File

@ -5,23 +5,23 @@ import (
"fmt"
"slices"
"github.com/foomo/contentserver/client"
contentserverclient "github.com/foomo/contentserver/client"
"github.com/foomo/contentserver/content"
typesense "github.com/foomo/typesense/pkg"
pkgx "github.com/foomo/typesense/pkg"
"go.uber.org/zap"
)
type ContentServer[indexDocument any] struct {
l *zap.Logger
contentserverClient *client.Client
documentProviderFuncs map[typesense.DocumentType]typesense.DocumentProviderFunc[indexDocument]
contentserverClient *contentserverclient.Client
documentProviderFuncs map[pkgx.DocumentType]pkgx.DocumentProviderFunc[indexDocument]
supportedMimeTypes []string
}
func NewContentServer[indexDocument any](
l *zap.Logger,
client *client.Client,
documentProviderFuncs map[typesense.DocumentType]typesense.DocumentProviderFunc[indexDocument],
client *contentserverclient.Client,
documentProviderFuncs map[pkgx.DocumentType]pkgx.DocumentProviderFunc[indexDocument],
supportedMimeTypes []string,
) *ContentServer[indexDocument] {
return &ContentServer[indexDocument]{
@ -34,7 +34,7 @@ func NewContentServer[indexDocument any](
func (c ContentServer[indexDocument]) Provide(
ctx context.Context,
indexID typesense.IndexID,
indexID pkgx.IndexID,
) ([]*indexDocument, error) {
documentInfos, err := c.getDocumentIDsByIndexID(ctx, indexID)
if err != nil {
@ -71,7 +71,7 @@ func (c ContentServer[indexDocument]) Provide(
func (c ContentServer[indexDocument]) ProvidePaged(
ctx context.Context,
indexID typesense.IndexID,
indexID pkgx.IndexID,
offset int,
) ([]*indexDocument, int, error) {
panic("implement me")
@ -79,8 +79,8 @@ func (c ContentServer[indexDocument]) ProvidePaged(
func (c ContentServer[indexDocument]) getDocumentIDsByIndexID(
ctx context.Context,
indexID typesense.IndexID,
) ([]typesense.DocumentInfo, error) {
indexID pkgx.IndexID,
) ([]pkgx.DocumentInfo, error) {
// get the contentserver dimension defined by indexID
// create the list of document infos
repo, err := c.contentserverClient.GetRepo(ctx)
@ -93,12 +93,12 @@ func (c ContentServer[indexDocument]) getDocumentIDsByIndexID(
}
nodeMap := createFlatRepoNodeMap(rootRepoNode, map[string]*content.RepoNode{})
documentInfos := make([]typesense.DocumentInfo, 0, len(nodeMap))
documentInfos := make([]pkgx.DocumentInfo, 0, len(nodeMap))
for _, repoNode := range nodeMap {
if slices.Contains(c.supportedMimeTypes, repoNode.MimeType) {
documentInfos = append(documentInfos, typesense.DocumentInfo{
DocumentType: typesense.DocumentType(repoNode.MimeType),
DocumentID: typesense.DocumentID(repoNode.ID),
documentInfos = append(documentInfos, pkgx.DocumentInfo{
DocumentType: pkgx.DocumentType(repoNode.MimeType),
DocumentID: pkgx.DocumentID(repoNode.ID),
})
}
}
@ -122,9 +122,9 @@ func createFlatRepoNodeMap(node *content.RepoNode, nodeMap map[string]*content.R
func (c ContentServer[indexDocument]) fetchURLsByDocumentIDs(
ctx context.Context,
indexID typesense.IndexID,
documentInfos []typesense.DocumentInfo,
) (map[typesense.DocumentID]string, error) {
indexID pkgx.IndexID,
documentInfos []pkgx.DocumentInfo,
) (map[pkgx.DocumentID]string, error) {
ids := make([]string, len(documentInfos))
for i, documentInfo := range documentInfos {
@ -140,10 +140,10 @@ func (c ContentServer[indexDocument]) fetchURLsByDocumentIDs(
return convertMapStringToDocumentID(uriMap), nil
}
func convertMapStringToDocumentID(input map[string]string) map[typesense.DocumentID]string {
output := make(map[typesense.DocumentID]string, len(input))
func convertMapStringToDocumentID(input map[string]string) map[pkgx.DocumentID]string {
output := make(map[pkgx.DocumentID]string, len(input))
for key, value := range input {
output[typesense.DocumentID(key)] = value
output[pkgx.DocumentID(key)] = value
}
return output
}

View File

@ -3,20 +3,20 @@ package typesenseindexing
import (
"context"
typesense "github.com/foomo/typesense/pkg"
pkgx "github.com/foomo/typesense/pkg"
"go.uber.org/zap"
)
type BaseIndexer[indexDocument any, returnType any] struct {
l *zap.Logger
typesenseAPI typesense.API[indexDocument, returnType]
documentProvider typesense.DocumentProvider[indexDocument]
typesenseAPI pkgx.API[indexDocument, returnType]
documentProvider pkgx.DocumentProvider[indexDocument]
}
func NewBaseIndexer[indexDocument any, returnType any](
l *zap.Logger,
typesenseAPI typesense.API[indexDocument, returnType],
documentProvider typesense.DocumentProvider[indexDocument],
typesenseAPI pkgx.API[indexDocument, returnType],
documentProvider pkgx.DocumentProvider[indexDocument],
) *BaseIndexer[indexDocument, returnType] {
return &BaseIndexer[indexDocument, returnType]{
l: l,
@ -33,14 +33,14 @@ func (b *BaseIndexer[indexDocument, returnType]) Run(ctx context.Context) error
// Step 1: Ensure Typesense is initialized
revisionID, err := b.typesenseAPI.Initialize(ctx)
if err != nil || revisionID == "" {
b.l.Error("Failed to initialize Typesense", zap.Error(err))
b.l.Error("failed to initialize typesense", zap.Error(err))
return err
}
// Step 2: Retrieve all configured indices
indices, err := b.typesenseAPI.Indices()
if err != nil {
b.l.Error("Failed to retrieve indices from Typesense", zap.Error(err))
b.l.Error("failed to retrieve indices from typesense", zap.Error(err))
return err
}
@ -52,7 +52,7 @@ func (b *BaseIndexer[indexDocument, returnType]) Run(ctx context.Context) error
// Fetch documents from the provider
documents, err := b.documentProvider.Provide(ctx, indexID)
if err != nil {
b.l.Error("Failed to fetch documents", zap.String("index", string(indexID)), zap.Error(err))
b.l.Error("failed to fetch documents", zap.String("index", string(indexID)), zap.Error(err))
tainted = true
continue
}
@ -60,7 +60,7 @@ func (b *BaseIndexer[indexDocument, returnType]) Run(ctx context.Context) error
err = b.typesenseAPI.UpsertDocuments(ctx, revisionID, indexID, documents)
if err != nil {
b.l.Error(
"Failed to upsert documents",
"failed to upsert documents",
zap.String("index", string(indexID)),
zap.String("revision", string(revisionID)),
zap.Int("documents", len(documents)),
@ -71,7 +71,7 @@ func (b *BaseIndexer[indexDocument, returnType]) Run(ctx context.Context) error
}
indexedDocuments += len(documents)
b.l.Info("Successfully upserted documents",
b.l.Info("successfully upserted documents",
zap.String("index", string(indexID)),
zap.Int("count", len(documents)),
)
@ -82,20 +82,20 @@ func (b *BaseIndexer[indexDocument, returnType]) Run(ctx context.Context) error
// No errors encountered, commit the revision
err = b.typesenseAPI.CommitRevision(ctx, revisionID)
if err != nil {
b.l.Error("Failed to commit revision", zap.String("revision", string(revisionID)), zap.Error(err))
b.l.Error("failed to commit revision", zap.String("revision", string(revisionID)), zap.Error(err))
return err
}
b.l.Info("Successfully committed revision", zap.String("revision", string(revisionID)))
b.l.Info("successfully committed revision", zap.String("revision", string(revisionID)))
} else {
// If errors occurred, revert the revision
b.l.Warn("Errors detected during upsert, reverting revision", zap.String("revision", string(revisionID)))
b.l.Warn("errors detected during upsert, reverting revision", zap.String("revision", string(revisionID)))
err = b.typesenseAPI.RevertRevision(ctx, revisionID)
if err != nil {
b.l.Error("Failed to revert revision", zap.String("revision", string(revisionID)), zap.Error(err))
b.l.Error("failed to revert revision", zap.String("revision", string(revisionID)), zap.Error(err))
return err
}
b.l.Info("Successfully reverted revision", zap.String("revision", string(revisionID)))
b.l.Info("successfully reverted revision", zap.String("revision", string(revisionID)))
}
return nil