mirror of
https://github.com/foomo/typesense.git
synced 2025-10-16 12:45:37 +00:00
commit
4a4d334336
133
.github/CODE_OF_CONDUCT.md
vendored
Normal file
133
.github/CODE_OF_CONDUCT.md
vendored
Normal file
@ -0,0 +1,133 @@
|
||||
|
||||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
We as members, contributors, and leaders pledge to make participation in our
|
||||
community a harassment-free experience for everyone, regardless of age, body
|
||||
size, visible or invisible disability, ethnicity, sex characteristics, gender
|
||||
identity and expression, level of experience, education, socio-economic status,
|
||||
nationality, personal appearance, race, caste, color, religion, or sexual
|
||||
identity and orientation.
|
||||
|
||||
We pledge to act and interact in ways that contribute to an open, welcoming,
|
||||
diverse, inclusive, and healthy community.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to a positive environment for our
|
||||
community include:
|
||||
|
||||
* Demonstrating empathy and kindness toward other people
|
||||
* Being respectful of differing opinions, viewpoints, and experiences
|
||||
* Giving and gracefully accepting constructive feedback
|
||||
* Accepting responsibility and apologizing to those affected by our mistakes,
|
||||
and learning from the experience
|
||||
* Focusing on what is best not just for us as individuals, but for the overall
|
||||
community
|
||||
|
||||
Examples of unacceptable behavior include:
|
||||
|
||||
* The use of sexualized language or imagery, and sexual attention or advances of
|
||||
any kind
|
||||
* Trolling, insulting or derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or email address,
|
||||
without their explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Enforcement Responsibilities
|
||||
|
||||
Community leaders are responsible for clarifying and enforcing our standards of
|
||||
acceptable behavior and will take appropriate and fair corrective action in
|
||||
response to any behavior that they deem inappropriate, threatening, offensive,
|
||||
or harmful.
|
||||
|
||||
Community leaders have the right and responsibility to remove, edit, or reject
|
||||
comments, commits, code, wiki edits, issues, and other contributions that are
|
||||
not aligned to this Code of Conduct, and will communicate reasons for moderation
|
||||
decisions when appropriate.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies within all community spaces, and also applies when
|
||||
an individual is officially representing the community in public spaces.
|
||||
Examples of representing our community include using an official email address,
|
||||
posting via an official social media account, or acting as an appointed
|
||||
representative at an online or offline event.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported to the community leaders responsible for enforcement at
|
||||
info@bestbytes.com.
|
||||
All complaints will be reviewed and investigated promptly and fairly.
|
||||
|
||||
All community leaders are obligated to respect the privacy and security of the
|
||||
reporter of any incident.
|
||||
|
||||
## Enforcement Guidelines
|
||||
|
||||
Community leaders will follow these Community Impact Guidelines in determining
|
||||
the consequences for any action they deem in violation of this Code of Conduct:
|
||||
|
||||
### 1. Correction
|
||||
|
||||
**Community Impact**: Use of inappropriate language or other behavior deemed
|
||||
unprofessional or unwelcome in the community.
|
||||
|
||||
**Consequence**: A private, written warning from community leaders, providing
|
||||
clarity around the nature of the violation and an explanation of why the
|
||||
behavior was inappropriate. A public apology may be requested.
|
||||
|
||||
### 2. Warning
|
||||
|
||||
**Community Impact**: A violation through a single incident or series of
|
||||
actions.
|
||||
|
||||
**Consequence**: A warning with consequences for continued behavior. No
|
||||
interaction with the people involved, including unsolicited interaction with
|
||||
those enforcing the Code of Conduct, for a specified period of time. This
|
||||
includes avoiding interactions in community spaces as well as external channels
|
||||
like social media. Violating these terms may lead to a temporary or permanent
|
||||
ban.
|
||||
|
||||
### 3. Temporary Ban
|
||||
|
||||
**Community Impact**: A serious violation of community standards, including
|
||||
sustained inappropriate behavior.
|
||||
|
||||
**Consequence**: A temporary ban from any sort of interaction or public
|
||||
communication with the community for a specified period of time. No public or
|
||||
private interaction with the people involved, including unsolicited interaction
|
||||
with those enforcing the Code of Conduct, is allowed during this period.
|
||||
Violating these terms may lead to a permanent ban.
|
||||
|
||||
### 4. Permanent Ban
|
||||
|
||||
**Community Impact**: Demonstrating a pattern of violation of community
|
||||
standards, including sustained inappropriate behavior, harassment of an
|
||||
individual, or aggression toward or disparagement of classes of individuals.
|
||||
|
||||
**Consequence**: A permanent ban from any sort of public interaction within the
|
||||
community.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
|
||||
version 2.1, available at
|
||||
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
|
||||
|
||||
Community Impact Guidelines were inspired by
|
||||
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
|
||||
|
||||
For answers to common questions about this code of conduct, see the FAQ at
|
||||
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
|
||||
[https://www.contributor-covenant.org/translations][translations].
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
|
||||
[Mozilla CoC]: https://github.com/mozilla/diversity
|
||||
[FAQ]: https://www.contributor-covenant.org/faq
|
||||
[translations]: https://www.contributor-covenant.org/translations
|
||||
21
.github/CONTRIBUTING.md
vendored
Normal file
21
.github/CONTRIBUTING.md
vendored
Normal file
@ -0,0 +1,21 @@
|
||||
# Contributing
|
||||
|
||||
If you want to submit a pull request to fix a bug or enhance an existing
|
||||
feature, please first open an issue and link to that issue when you
|
||||
submit your pull request.
|
||||
|
||||
If you have any questions about a possible submission, feel free to open
|
||||
an issue too.
|
||||
|
||||
### Pull request process
|
||||
|
||||
1. Fork this repository
|
||||
2. Create a branch in your fork to implement the changes. We recommend using
|
||||
the issue number as part of your branch name, e.g. `1234-fixes`
|
||||
3. Ensure that any documentation is updated with the changes that are required
|
||||
by your fix.
|
||||
4. Ensure that any samples are updated if the base image has been changed.
|
||||
5. Submit the pull request. *Do not leave the pull request blank*. Explain exactly
|
||||
what your changes are meant to do and provide simple steps on how to validate
|
||||
your changes. Ensure that you reference the issue you created as well.
|
||||
The pull request will be review before it is merged.
|
||||
17
.github/ISSUE_TEMPLATE/bug-report.md
vendored
Normal file
17
.github/ISSUE_TEMPLATE/bug-report.md
vendored
Normal file
@ -0,0 +1,17 @@
|
||||
---
|
||||
name: Bug Report
|
||||
about: Report a bug you encountered
|
||||
labels: bug
|
||||
---
|
||||
**What happened**:
|
||||
|
||||
**What you expected to happen**:
|
||||
|
||||
**How to reproduce it (as minimally and precisely as possible)**:
|
||||
|
||||
**Anything else we need to know?**:
|
||||
|
||||
**Environment**:
|
||||
- Affected Version:
|
||||
- OS (e.g: `cat /etc/os-release`):
|
||||
- Others:
|
||||
8
.github/ISSUE_TEMPLATE/enhancement.md
vendored
Normal file
8
.github/ISSUE_TEMPLATE/enhancement.md
vendored
Normal file
@ -0,0 +1,8 @@
|
||||
---
|
||||
name: Enhancement Request
|
||||
about: Suggest an enhancement
|
||||
labels: enhancement
|
||||
---
|
||||
**What would you like to be added**:
|
||||
|
||||
**Why is this needed**:
|
||||
19
.github/PULL_REQUEST_TEMPLATE.md
vendored
Normal file
19
.github/PULL_REQUEST_TEMPLATE.md
vendored
Normal file
@ -0,0 +1,19 @@
|
||||
### Type of Change
|
||||
- [ ] New feature
|
||||
- [ ] Bug fix
|
||||
- [ ] Documentation update
|
||||
- [ ] Refactoring
|
||||
- [ ] Hotfix
|
||||
- [ ] Security patch
|
||||
|
||||
### Description
|
||||
_[Provide a detailed explanation of the changes you have made. Include the reasons behind these changes and any relevant context. Link any related issues.]_
|
||||
|
||||
### Related Issues
|
||||
_[If this pull request addresses an issue, please link to it here (e.g., Fixes #123).]_
|
||||
|
||||
### Checklist
|
||||
- [ ] My code adheres to the coding and style guidelines of the project.
|
||||
- [ ] I have performed a self-review of my own code.
|
||||
- [ ] I have commented my code, particularly in hard-to-understand areas.
|
||||
- [ ] I have made corresponding changes to the documentation.
|
||||
45
.github/SECURITY.md
vendored
Normal file
45
.github/SECURITY.md
vendored
Normal file
@ -0,0 +1,45 @@
|
||||
# Security Guidelines
|
||||
|
||||
## How security is managed on this project
|
||||
|
||||
The foomo team and community take security seriously and wants to ensure that
|
||||
we maintain a secure environment and provide secure solutions for the open
|
||||
source community. To help us achieve these goals, please note the
|
||||
following before using this software:
|
||||
|
||||
- Review the software license to understand the contributor's obligations in
|
||||
terms of warranties and suitability for purpose
|
||||
- For any questions or concerns about security, you can
|
||||
[create an issue][new-issue] or [report a vulnerability][new-sec-issue]
|
||||
- We request that you work with our security team and opt for
|
||||
responsible disclosure using the guidelines below
|
||||
- All security related issues and pull requests you make should be tagged with
|
||||
"security" for easy identification
|
||||
- Please monitor this repository and update your environment in a timely manner
|
||||
as we release patches and updates
|
||||
|
||||
## Responsibly Disclosing Security Bugs
|
||||
|
||||
If you find a security bug in this repository, please work with contributors
|
||||
following responsible disclosure principles and these guidelines:
|
||||
|
||||
- Do not submit a normal issue or pull request in our public repository, instead
|
||||
[report it directly][new-sec-issue].
|
||||
- We will review your submission and may follow up for additional details
|
||||
- If you have a patch, we will review it and approve it privately; once approved
|
||||
for release you can submit it as a pull request publicly in the repository (we
|
||||
give credit where credit is due)
|
||||
- We will keep you informed during our investigation, feel free to check in for
|
||||
a status update
|
||||
- We will release the fix and publicly disclose the issue as soon as possible,
|
||||
but want to ensure we due properly due diligence before releasing
|
||||
- Please do not publicly blog or post about the security issue until after we
|
||||
have updated the public repo so that other downstream users have an opportunity
|
||||
to patch
|
||||
|
||||
## Contact / Misc
|
||||
|
||||
If you have any questions, please reach out directly by [creating an issue][new-issue].
|
||||
|
||||
[new-issue]: https://github.com/foomo/typesense/issues/new/choose
|
||||
[new-sec-issue]: https://github.com/foomo/typesense/security/advisories/new
|
||||
31
.github/dependabot.yml
vendored
Normal file
31
.github/dependabot.yml
vendored
Normal file
@ -0,0 +1,31 @@
|
||||
# Please see the documentation for all configuration options:
|
||||
# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
|
||||
version: 2
|
||||
|
||||
updates:
|
||||
- package-ecosystem: 'github-actions'
|
||||
directory: '/'
|
||||
schedule:
|
||||
day: 'sunday'
|
||||
interval: 'weekly'
|
||||
groups:
|
||||
github-actions:
|
||||
patterns: ['*']
|
||||
|
||||
- package-ecosystem: 'gomod'
|
||||
directory: '/'
|
||||
schedule:
|
||||
day: 'sunday'
|
||||
interval: 'weekly'
|
||||
groups:
|
||||
gomod-security:
|
||||
applies-to: security-updates
|
||||
update-types: ['minor', 'patch']
|
||||
patterns: ['*']
|
||||
gomod-update:
|
||||
applies-to: version-updates
|
||||
update-types: ['minor', 'patch']
|
||||
patterns: ['*']
|
||||
ignore:
|
||||
- dependency-name: "*"
|
||||
update-types: ["version-update:semver-major"]
|
||||
30
.github/workflows/release.yml
vendored
Normal file
30
.github/workflows/release.yml
vendored
Normal file
@ -0,0 +1,30 @@
|
||||
name: Release Tag
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- v*.*.*
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
release:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
check-latest: true
|
||||
go-version-file: go.mod
|
||||
|
||||
- uses: goreleaser/goreleaser-action@v6
|
||||
with:
|
||||
version: latest
|
||||
args: release --clean
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
39
.github/workflows/test.yml
vendored
Normal file
39
.github/workflows/test.yml
vendored
Normal file
@ -0,0 +1,39 @@
|
||||
name: Test Branch
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ main ]
|
||||
pull_request:
|
||||
merge_group:
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: "${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}"
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
check-latest: true
|
||||
go-version-file: 'go.mod'
|
||||
|
||||
- uses: gotesttools/gotestfmt-action@v2
|
||||
with:
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- uses: golangci/golangci-lint-action@v6
|
||||
with:
|
||||
version: latest
|
||||
args: --timeout=5m
|
||||
|
||||
- name: Run tests
|
||||
run: GO_TEST_TAGS=-skip go test -coverprofile=coverage.out -race -json ./... | gotestfmt
|
||||
|
||||
#- uses: coverallsapp/github-action@v2
|
||||
# with:
|
||||
# file: coverage.out
|
||||
40
.gitignore
vendored
40
.gitignore
vendored
@ -1,25 +1,25 @@
|
||||
# If you prefer the allow list template instead of the deny list, see community template:
|
||||
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
|
||||
#
|
||||
# Binaries for programs and plugins
|
||||
*.exe
|
||||
*.exe~
|
||||
*.dll
|
||||
*.so
|
||||
*.dylib
|
||||
|
||||
# Test binary, built with `go test -c`
|
||||
*.test
|
||||
|
||||
# Output of the go coverage tool, specifically when used with LiteIDE
|
||||
.*
|
||||
*.log
|
||||
*.out
|
||||
|
||||
# Dependency directories (remove the comment below to include it)
|
||||
# vendor/
|
||||
/bin/
|
||||
/tmp/
|
||||
|
||||
# Go workspace file
|
||||
## Git
|
||||
!.gitignore
|
||||
|
||||
## GitHub
|
||||
!.github/
|
||||
|
||||
## Husky
|
||||
!.husky/
|
||||
!.husky.yaml
|
||||
|
||||
## EditorConfig
|
||||
!.editorconfig
|
||||
|
||||
## Go
|
||||
!.golangci.yml
|
||||
!.goreleaser.yml
|
||||
go.work
|
||||
go.work.sum
|
||||
|
||||
# env file
|
||||
.env
|
||||
|
||||
165
.golangci.yml
165
.golangci.yml
@ -1,5 +1,10 @@
|
||||
# yaml-language-server: $schema=https://golangci-lint.run/jsonschema/golangci.jsonschema.json
|
||||
# https://golangci-lint.run/usage/configuration/
|
||||
|
||||
run:
|
||||
timeout: 5m
|
||||
go: 1.24.1
|
||||
build-tags: [safe]
|
||||
modules-download-mode: readonly
|
||||
|
||||
linters-settings:
|
||||
importas:
|
||||
@ -7,54 +12,148 @@ linters-settings:
|
||||
- pkg: '^github.com\/foomo\/typesense\/(.*\/)?([^\/]+)\/?$'
|
||||
alias: '${2}x' # enforce `x` suffix
|
||||
no-unaliased: true
|
||||
# https://golangci-lint.run/usage/linters/#revive
|
||||
revive:
|
||||
ignore-generated-header: true
|
||||
enable-all-rules: true
|
||||
rules:
|
||||
- name: unused-parameter
|
||||
disabled: true
|
||||
- name: line-length-limit
|
||||
disabled: true
|
||||
- name: add-constant
|
||||
disabled: true
|
||||
- name: function-result-limit
|
||||
disabled: true
|
||||
- name: cognitive-complexity
|
||||
disabled: true
|
||||
- name: unused-receiver
|
||||
disabled: true
|
||||
- name: use-any
|
||||
disabled: true
|
||||
- name: cyclomatic
|
||||
disabled: true
|
||||
- name: function-length
|
||||
disabled: true
|
||||
- name: max-public-structs
|
||||
disabled: true
|
||||
|
||||
linters:
|
||||
disable-all: true
|
||||
enable:
|
||||
# Enabled by default linters:
|
||||
- errcheck
|
||||
- gosimple
|
||||
- govet
|
||||
- ineffassign
|
||||
- staticcheck
|
||||
## Default linters
|
||||
- errcheck # errcheck is a program for checking for unchecked errors in Go code. These unchecked errors can be critical bugs in some cases [fast: false, auto-fix: false]
|
||||
- gosimple # (megacheck) Linter for Go source code that specializes in simplifying code [fast: false, auto-fix: false]
|
||||
- govet # (vet, vetshadow) Vet examines Go source code and reports suspicious constructs. It is roughly the same as 'go vet' and uses its passes. [fast: false, auto-fix: false]
|
||||
- ineffassign # Detects when assignments to existing variables are not used [fast: true, auto-fix: false]
|
||||
- staticcheck # (megacheck) It's a set of rules from staticcheck. It's not the same thing as the staticcheck binary. The author of staticcheck doesn't support or approve the use of staticcheck as a library inside golangci-lint. [fast: false, auto-fix: false]
|
||||
- unused # (megacheck) Checks Go code for unused constants, variables, functions and types [fast: false, auto-fix: false]
|
||||
|
||||
# Disabled by default linters:
|
||||
- asciicheck # Simple linter to check that your code does not contain non-ASCII identifiers [fast: true, auto-fix: false]
|
||||
## Recommended linters
|
||||
- asasalint # check for pass []any as any in variadic func(...any) [fast: false, auto-fix: false]
|
||||
- asciicheck # checks that all code identifiers does not have non-ASCII symbols in the name [fast: true, auto-fix: false]
|
||||
- bidichk # Checks for dangerous unicode character sequences [fast: true, auto-fix: false]
|
||||
# - depguard # Go linter that checks if package imports are in a list of acceptable packages [fast: true, auto-fix: false]
|
||||
# - dupl # Tool for code clone detection [fast: true, auto-fix: false]
|
||||
- bodyclose # checks whether HTTP response body is closed successfully [fast: false, auto-fix: false]
|
||||
- canonicalheader # checks whether net/http.Header uses canonical header [fast: false, auto-fix: false]
|
||||
- containedctx # containedctx is a linter that detects struct contained context.Context field [fast: false, auto-fix: false]
|
||||
- contextcheck # check whether the function uses a non-inherited context [fast: false, auto-fix: false]
|
||||
- copyloopvar # (go >= 1.22) copyloopvar is a linter detects places where loop variables are copied [fast: true, auto-fix: false]
|
||||
- decorder # check declaration order and count of types, constants, variables and functions [fast: true, auto-fix: false]
|
||||
- durationcheck # check for two durations multiplied together [fast: false, auto-fix: false]
|
||||
- errchkjson # Checks types passed to the json encoding functions. Reports unsupported types and reports occations, where the check for the returned error can be omitted. [fast: false, auto-fix: false]
|
||||
- errname # Checks that sentinel errors are prefixed with the `Err` and error types are suffixed with the `Error`. [fast: false, auto-fix: false]
|
||||
- errorlint # errorlint is a linter for that can be used to find code that will cause problems with the error wrapping scheme introduced in Go 1.13. [fast: false, auto-fix: false]
|
||||
- exhaustive # check exhaustiveness of enum switch statements [fast: false, auto-fix: false]
|
||||
- exptostd # Detects functions from golang.org/x/exp/ that can be replaced by std functions. [auto-fix]
|
||||
- fatcontext # detects nested contexts in loops and function literals [fast: false, auto-fix: false]
|
||||
- forbidigo # Forbids identifiers [fast: false, auto-fix: false]
|
||||
- forcetypeassert # finds forced type assertions [fast: true, auto-fix: false]
|
||||
- gochecknoinits # Checks that no init functions are present in Go code [fast: true, auto-fix: false]
|
||||
- gocheckcompilerdirectives # Checks that go compiler directive comments (//go:) are valid. [fast: true, auto-fix: false]
|
||||
- gochecksumtype # Run exhaustiveness checks on Go "sum types" [fast: false, auto-fix: false]
|
||||
- goconst # Finds repeated strings that could be replaced by a constant [fast: true, auto-fix: false]
|
||||
- gocritic # Provides diagnostics that check for bugs, performance and style issues. [fast: false, auto-fix: false]
|
||||
- goimports # In addition to fixing imports, goimports also formats your code in the same style as gofmt. [fast: true, auto-fix: true]
|
||||
- gocritic # Provides diagnostics that check for bugs, performance and style issues. [fast: false, auto-fix: true]
|
||||
- gofmt # Gofmt checks whether code was gofmt-ed. By default this tool runs with -s option to check for code simplification [fast: true, auto-fix: true]
|
||||
- goheader # Checks is file header matches to pattern [fast: true, auto-fix: true]
|
||||
- goimports # Check import statements are formatted according to the 'goimport' command. Reformat imports in autofix mode. [fast: true, auto-fix: true]
|
||||
- gomoddirectives # Manage the use of 'replace', 'retract', and 'excludes' directives in go.mod. [fast: true, auto-fix: false]
|
||||
- gomodguard # Allow and block list linter for direct Go module dependencies. This is different from depguard where there are different block types for example version constraints and module recommendations. [fast: true, auto-fix: false]
|
||||
- goprintffuncname # Checks that printf-like functions are named with `f` at the end [fast: true, auto-fix: false]
|
||||
- gosec # (gas): Inspects source code for security problems [fast: false, auto-fix: false]
|
||||
- grouper # An analyzer to analyze expression groups. [fast: true, auto-fix: false]
|
||||
# - importas # Enforces consistent import aliases [fast: false, auto-fix: false]
|
||||
- maintidx # maintidx measures the maintainability index of each function. [fast: true, auto-fix: false]
|
||||
- goprintffuncname # Checks that printf-like functions are named with `f` at the end. [fast: true, auto-fix: false]
|
||||
- gosec # (gas) Inspects source code for security problems [fast: false, auto-fix: false]
|
||||
- gosmopolitan # Report certain i18n/l10n anti-patterns in your Go codebase [fast: false, auto-fix: false]
|
||||
- grouper # Analyze expression groups. [fast: true, auto-fix: false]
|
||||
- iface # Detect the incorrect use of interfaces, helping developers avoid interface pollution. [fast: false, auto-fix: false]
|
||||
- importas # Enforces consistent import aliases [fast: false, auto-fix: false]
|
||||
- inamedparam # reports interfaces with unnamed method parameters [fast: true, auto-fix: false]
|
||||
- intrange # (go >= 1.22) intrange is a linter to find places where for loops could make use of an integer range. [fast: true, auto-fix: false]
|
||||
- loggercheck # (logrlint) Checks key value pairs for common logger libraries (kitlog,klog,logr,zap). [fast: false, auto-fix: false]
|
||||
- makezero # Finds slice declarations with non-zero initial length [fast: false, auto-fix: false]
|
||||
- misspell # Finds commonly misspelled English words in comments [fast: true, auto-fix: true]
|
||||
- nakedret # Finds naked returns in functions greater than a specified function length [fast: true, auto-fix: false]
|
||||
- nestif # Reports deeply nested if statements [fast: true, auto-fix: false]
|
||||
- mirror # reports wrong mirror patterns of bytes/strings usage [fast: false, auto-fix: true]
|
||||
- misspell # Finds commonly misspelled English words [fast: true, auto-fix: true]
|
||||
- musttag # enforce field tags in (un)marshaled structs [fast: false, auto-fix: false]
|
||||
- nakedret # Checks that functions with naked returns are not longer than a maximum size (can be zero). [fast: true, auto-fix: false]
|
||||
- nilerr # Finds the code that returns nil even if it checks that the error is not nil. [fast: false, auto-fix: false]
|
||||
- nilnesserr # Reports constructs that checks for err != nil, but returns a different nil value error.
|
||||
- nilnil # Checks that there is no simultaneous return of `nil` error and an invalid value. [fast: false, auto-fix: false]
|
||||
- noctx # noctx finds sending http request without context.Context [fast: false, auto-fix: false]
|
||||
- nolintlint # Reports ill-formed or insufficient nolint directives [fast: true, auto-fix: false]
|
||||
# - nonamedreturns # Reports all named returns [fast: false, auto-fix: false]
|
||||
- noctx # Finds sending http request without context.Context [fast: false, auto-fix: false]
|
||||
- nolintlint # Reports ill-formed or insufficient nolint directives [fast: true, auto-fix: true]
|
||||
- nonamedreturns # Reports all named returns [fast: false, auto-fix: false]
|
||||
- nosprintfhostport # Checks for misuse of Sprintf to construct a host with port in a URL. [fast: true, auto-fix: false]
|
||||
- prealloc # Finds slice declarations that could potentially be pre-allocated [fast: true, auto-fix: false]
|
||||
- paralleltest # Detects missing usage of t.Parallel() method in your Go test [fast: false, auto-fix: false]
|
||||
- predeclared # find code that shadows one of Go's predeclared identifiers [fast: true, auto-fix: false]
|
||||
- promlinter # Check Prometheus metrics naming via promlint [fast: true, auto-fix: false]
|
||||
- reassign # Checks that package variables are not reassigned [fast: false, auto-fix: false]
|
||||
- recvcheck # checks for receiver type consistency [fast: false, auto-fix: false]
|
||||
- revive # Fast, configurable, extensible, flexible, and beautiful linter for Go. Drop-in replacement of golint. [fast: false, auto-fix: false]
|
||||
- tagliatelle # Checks the struct tags. [fast: true, auto-fix: false]
|
||||
- rowserrcheck # checks whether Rows.Err of rows is checked successfully [fast: false, auto-fix: false]
|
||||
- spancheck # Checks for mistakes with OpenTelemetry/Census spans. [fast: false, auto-fix: false]
|
||||
- sqlclosecheck # Checks that sql.Rows, sql.Stmt, sqlx.NamedStmt, pgx.Query are closed. [fast: false, auto-fix: false]
|
||||
- stylecheck # Stylecheck is a replacement for golint [fast: false, auto-fix: false]
|
||||
- testableexamples # linter checks if examples are testable (have an expected output) [fast: true, auto-fix: false]
|
||||
- testifylint # Checks usage of github.com/stretchr/testify. [fast: false, auto-fix: false]
|
||||
- testpackage # linter that makes you use a separate _test package [fast: true, auto-fix: false]
|
||||
- thelper # thelper detects golang test helpers without t.Helper() call and checks the consistency of test helpers [fast: false, auto-fix: false]
|
||||
- thelper # thelper detects tests helpers which is not start with t.Helper() method. [fast: false, auto-fix: false]
|
||||
- tparallel # tparallel detects inappropriate usage of t.Parallel() method in your Go test codes. [fast: false, auto-fix: false]
|
||||
- unconvert # Remove unnecessary type conversions [fast: false, auto-fix: false]
|
||||
- unparam # Reports unused function parameters [fast: false, auto-fix: false]
|
||||
- usestdlibvars # A linter that detect the possibility to use variables/constants from the Go standard library. [fast: true, auto-fix: false]
|
||||
- wastedassign # wastedassign finds wasted assignment statements. [fast: false, auto-fix: false]
|
||||
- whitespace # Tool for detection of leading and trailing whitespace [fast: true, auto-fix: true]
|
||||
disable:
|
||||
- unused
|
||||
- usetesting # Reports uses of functions with replacement inside the testing package. [auto-fix]
|
||||
- wastedassign # Finds wasted assignment statements [fast: false, auto-fix: false]
|
||||
- whitespace # Whitespace is a linter that checks for unnecessary newlines at the start and end of functions, if, for, etc. [fast: true, auto-fix: true]
|
||||
|
||||
## Discouraged linters
|
||||
#- cyclop # checks function and package cyclomatic complexity [fast: false, auto-fix: false]
|
||||
#- depguard # Go linter that checks if package imports are in a list of acceptable packages [fast: true, auto-fix: false]
|
||||
#- dogsled # Checks assignments with too many blank identifiers (e.g. x, _, _, _, := f()) [fast: true, auto-fix: false]
|
||||
#- dupl # Tool for code clone detection [fast: true, auto-fix: false]
|
||||
#- dupword # checks for duplicate words in the source code [fast: true, auto-fix: false]
|
||||
#- dogsled # Checks assignments with too many blank identifiers (e.g. x, _, _, _, := f()) [fast: true, auto-fix: false]
|
||||
#- err113 # Go linter to check the errors handling expressions [fast: false, auto-fix: false]
|
||||
#- exhaustruct # Checks if all structure fields are initialized [fast: false, auto-fix: false]
|
||||
#- funlen # Tool for detection of long functions [fast: true, auto-fix: false]
|
||||
#- gci # Gci controls Go package import order and makes it always deterministic. [fast: true, auto-fix: true]
|
||||
#- ginkgolinter # enforces standards of using ginkgo and gomega [fast: false, auto-fix: false]
|
||||
#- gochecknoglobals # Check that no global variables exist. [fast: false, auto-fix: false]
|
||||
#- gochecknoinits # Checks that no init functions are present in Go code [fast: true, auto-fix: false]
|
||||
#- gocognit # Computes and checks the cognitive complexity of functions [fast: true, auto-fix: false]
|
||||
#- gocyclo # Computes and checks the cyclomatic complexity of functions [fast: true, auto-fix: false]
|
||||
#- godot # Check if comments end in a period [fast: true, auto-fix: true]
|
||||
#- godox # Tool for detection of comment keywords [fast: true, auto-fix: false]
|
||||
#- gofumpt # Gofumpt checks whether code was gofumpt-ed. [fast: true, auto-fix: true]
|
||||
#- interfacebloat # A linter that checks the number of methods inside an interface. [fast: true, auto-fix: false]
|
||||
#- intrange # (go >= 1.22) intrange is a linter to find places where for loops could make use of an integer range. [fast: true, auto-fix: false]
|
||||
#- ireturn # Accept Interfaces, Return Concrete Types [fast: false, auto-fix: false]
|
||||
#- lll # Reports long lines [fast: true, auto-fix: false]
|
||||
#- maintidx # maintidx measures the maintainability index of each function. [fast: true, auto-fix: false]
|
||||
#- nestif # Reports deeply nested if statements [fast: true, auto-fix: false]
|
||||
#- nlreturn # nlreturn checks for a new line before return and branch statements to increase code clarity [fast: true, auto-fix: false]
|
||||
#- mnd # An analyzer to detect magic numbers. [fast: true, auto-fix: false]
|
||||
#- perfsprint # Checks that fmt.Sprintf can be replaced with a faster alternative. [fast: false, auto-fix: false]
|
||||
#- prealloc # Finds slice declarations that could potentially be pre-allocated [fast: true, auto-fix: false]
|
||||
#- protogetter # Reports direct reads from proto message fields when getters should be used [fast: false, auto-fix: true]
|
||||
#- sloglint # ensure consistent code style when using log/slog [fast: false, auto-fix: false]
|
||||
#- tagalign # check that struct tags are well aligned [fast: true, auto-fix: true]
|
||||
#- tagliatelle # Checks the struct tags. [fast: true, auto-fix: false]
|
||||
#- unparam # Reports unused function parameters [fast: false, auto-fix: false]
|
||||
#- varnamelen # checks that the length of a variable's name matches its scope [fast: false, auto-fix: false]
|
||||
#- wrapcheck # Checks that errors returned from external packages are wrapped [fast: false, auto-fix: false]
|
||||
#- wsl # add or remove empty lines [fast: true, auto-fix: false]
|
||||
#- zerologlint # Detects the wrong usage of `zerolog` that a user forgets to dispatch with `Send` or `Msg` [fast: false, auto-fix: false]
|
||||
|
||||
@ -1,34 +1,7 @@
|
||||
version: 2
|
||||
|
||||
builds:
|
||||
- skip: true
|
||||
|
||||
changelog:
|
||||
filters:
|
||||
exclude:
|
||||
- "^wip"
|
||||
- "^test"
|
||||
- "^docs"
|
||||
- "^chore"
|
||||
- "^style"
|
||||
- "go mod tidy"
|
||||
- "merge conflict"
|
||||
- "Merge pull request"
|
||||
- "Merge remote-tracking branch"
|
||||
- "Merge branch"
|
||||
groups:
|
||||
- title: Features
|
||||
regexp: '^.*?feat(\([[:word:]]+\))??!?:.+$'
|
||||
order: 0
|
||||
- title: Dependency updates
|
||||
regexp: '^.*?(feat|fix)\(deps\)!?:.+$'
|
||||
order: 100
|
||||
- title: "Bug fixes"
|
||||
regexp: '^.*?fix(\([[:word:]]+\))??!?:.+$'
|
||||
order: 150
|
||||
- title: "Security"
|
||||
regexp: '^.*?sec(\([[:word:]]+\))??!?:.+$'
|
||||
order: 200
|
||||
- title: "Performace"
|
||||
regexp: '^.*?perf(\([[:word:]]+\))??!?:.+$'
|
||||
order: 250
|
||||
- title: Other
|
||||
order: 999
|
||||
use: github-native
|
||||
|
||||
15
.husky.yaml
Normal file
15
.husky.yaml
Normal file
@ -0,0 +1,15 @@
|
||||
hooks:
|
||||
pre-commit:
|
||||
- golangci-lint run --fast
|
||||
- husky lint-staged
|
||||
commit-msg:
|
||||
# only execute if not in a merge
|
||||
- if [[ -z $(git rev-parse -q --verify MERGE_HEAD) ]]; then husky lint-commit; fi
|
||||
|
||||
lint-staged:
|
||||
'*.go':
|
||||
- goimports -l -w
|
||||
|
||||
lint-commit:
|
||||
types: '^(feat|fix|build|chore|docs|perf|refactor|revert|style|test|wip)$'
|
||||
header: '^(?P<type>\w+)(\((?P<scope>[\w/.-]+)\))?(?P<breaking>!)?:( +)?(?P<header>.+)'
|
||||
3
.husky/applypatch-msg
Executable file
3
.husky/applypatch-msg
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/commit-msg
Executable file
3
.husky/commit-msg
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/fsmonitor-watchman
Executable file
3
.husky/fsmonitor-watchman
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/post-update
Executable file
3
.husky/post-update
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/pre-applypatch
Executable file
3
.husky/pre-applypatch
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/pre-commit
Executable file
3
.husky/pre-commit
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/pre-merge-commit
Executable file
3
.husky/pre-merge-commit
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/pre-push
Executable file
3
.husky/pre-push
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/pre-rebase
Executable file
3
.husky/pre-rebase
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/pre-receive
Executable file
3
.husky/pre-receive
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/prepare-commit-msg
Executable file
3
.husky/prepare-commit-msg
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/push-to-checkout
Executable file
3
.husky/push-to-checkout
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/sendemail-validate
Executable file
3
.husky/sendemail-validate
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
3
.husky/update
Executable file
3
.husky/update
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
husky hook $(basename "$0") $*
|
||||
129
README.md
129
README.md
@ -0,0 +1,129 @@
|
||||
# Typesense API
|
||||
[](https://github.com/foomo/typesense/actions/workflows/test.yml)
|
||||
[](https://goreportcard.com/report/github.com/foomo/typesense)
|
||||
[](https://godoc.org/github.com/foomo/typesense)
|
||||
|
||||
## Overview
|
||||
This package provides an API for managing and searching Typesense collections. It offers functionalities for initializing, indexing, searching, and maintaining Typesense collections and aliases.
|
||||
|
||||
## Features
|
||||
- **Initialization**: Ensures that all aliases point to the latest revision-based collections.
|
||||
- **Health Check**: Verifies if the Typesense client is operational.
|
||||
- **Index Management**: Lists, creates, and updates index collections.
|
||||
- **Document Upsertion**: Bulk upsert support for indexing documents.
|
||||
- **Search Operations**: Provides simple and advanced search capabilities.
|
||||
- **Revision Management**: Supports committing and reverting indexing revisions.
|
||||
|
||||
## Installation
|
||||
To use this package, add it as a dependency in your Go project:
|
||||
```sh
|
||||
go get github.com/foomo/typesense
|
||||
```
|
||||
|
||||
## Usage example
|
||||
|
||||
```go
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/foomo/contentserver/client"
|
||||
"github.com/foomo/keel/config"
|
||||
"github.com/foomo/keel/log"
|
||||
typesenseapi "github.com/foomo/typesense/pkg/api"
|
||||
typesenseindexing "github.com/foomo/typesense/pkg/indexing"
|
||||
|
||||
"github.com/typesense/typesense-go/v3/typesense"
|
||||
)
|
||||
|
||||
func main() {
|
||||
ctx := context.Background()
|
||||
l := log.Logger()
|
||||
|
||||
// contentserver client
|
||||
csClient, errClient := client.NewHTTPClient("contentserver_url")
|
||||
log.Must(l, errClient, "could not get contentserver client")
|
||||
|
||||
// contentful client
|
||||
// provide list of content types
|
||||
cfClients := contentful.NewDefaultContentfulClients(ctx, l, contentful_types, true)
|
||||
cfClients.UpdateCache()
|
||||
cfClients.Client.ClientStats()
|
||||
|
||||
// create typesense client
|
||||
typesenseClient := typesense.NewClient(
|
||||
typesense.WithConnectionTimeout(2*time.Minute),
|
||||
typesense.WithServer("typesense_server"),
|
||||
typesense.WithAPIKey("typesense_api_key"),
|
||||
)
|
||||
|
||||
// configure document provider
|
||||
documentProvider := typesenseindexing.NewContentServer(
|
||||
l, csClient,
|
||||
GetDocumentProviderFunctions(...), // retrieve document provider functions
|
||||
supportedMimeTypes, // provide supported mime types
|
||||
)
|
||||
|
||||
// create typesense api
|
||||
// create indexDocument and returnType
|
||||
api := typesenseapi.NewBaseAPI[indexDocument, returnType](
|
||||
l,
|
||||
typesenseClient,
|
||||
collectionSchemas, //map[IndexID]*api.CollectionSchema
|
||||
presetUpsertSchema, //*api.PresetUpsertSchema
|
||||
)
|
||||
|
||||
// create typesense indexer
|
||||
indexer := typesenseindexing.NewBaseIndexer(
|
||||
l,
|
||||
api,
|
||||
documentProvider,
|
||||
)
|
||||
|
||||
// run indexer
|
||||
err := indexer.Run(ctx)
|
||||
log.Must(l, err, "could not run indexer")
|
||||
}
|
||||
```
|
||||
|
||||
### Health Check
|
||||
```go
|
||||
err := apiInstance.Healthz(context.Background())
|
||||
if err != nil {
|
||||
log.Fatalf("Health check failed: %v", err)
|
||||
}
|
||||
```
|
||||
|
||||
### Searching Documents
|
||||
#### Simple Search
|
||||
```go
|
||||
results, scores, total, err := apiInstance.SimpleSearch(context.Background(), "products", "laptop", nil, 1, 10, "price:desc")
|
||||
if err != nil {
|
||||
log.Fatalf("Search failed: %v", err)
|
||||
}
|
||||
log.Printf("Found %d results", total)
|
||||
```
|
||||
|
||||
#### Advanced Search
|
||||
```go
|
||||
searchParams := &api.SearchCollectionParams{
|
||||
Q: pointer.String("laptop"),
|
||||
SortBy: pointer.String("price:desc"),
|
||||
}
|
||||
|
||||
results, scores, total, err = apiInstance.ExpertSearch(context.Background(), "products", searchParams)
|
||||
if err != nil {
|
||||
log.Fatalf("Advanced search failed: %v", err)
|
||||
}
|
||||
log.Printf("Found %d results", total)
|
||||
```
|
||||
|
||||
## How to Contribute
|
||||
|
||||
Please refer to the [CONTRIBUTING](.github/CONTRIBUTING.md) details and follow the [CODE_OF_CONDUCT](.github/CODE_OF_CONDUCT.md) and [SECURITY](.github/SECURITY.md) guidelines.
|
||||
|
||||
## License
|
||||
|
||||
Distributed under MIT License, please see license file within the code for more details.
|
||||
|
||||
_Made with ♥ [foomo](https://www.foomo.org) by [bestbytes](https://www.bestbytes.com)_
|
||||
2
go.mod
2
go.mod
@ -1,6 +1,6 @@
|
||||
module github.com/foomo/typesense
|
||||
|
||||
go 1.24.0
|
||||
go 1.24.1
|
||||
|
||||
require (
|
||||
github.com/foomo/contentserver v1.11.2
|
||||
|
||||
@ -7,7 +7,7 @@ import (
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
pkgtypesense "github.com/foomo/typesense/pkg"
|
||||
pkgx "github.com/foomo/typesense/pkg"
|
||||
"github.com/typesense/typesense-go/v3/typesense"
|
||||
"github.com/typesense/typesense-go/v3/typesense/api"
|
||||
"github.com/typesense/typesense-go/v3/typesense/api/pointer"
|
||||
@ -19,16 +19,16 @@ const defaultSearchPresetName = "default"
|
||||
type BaseAPI[indexDocument any, returnType any] struct {
|
||||
l *zap.Logger
|
||||
client *typesense.Client
|
||||
collections map[pkgtypesense.IndexID]*api.CollectionSchema
|
||||
collections map[pkgx.IndexID]*api.CollectionSchema
|
||||
preset *api.PresetUpsertSchema
|
||||
|
||||
revisionID pkgtypesense.RevisionID
|
||||
revisionID pkgx.RevisionID
|
||||
}
|
||||
|
||||
func NewBaseAPI[indexDocument any, returnType any](
|
||||
l *zap.Logger,
|
||||
client *typesense.Client,
|
||||
collections map[pkgtypesense.IndexID]*api.CollectionSchema,
|
||||
collections map[pkgx.IndexID]*api.CollectionSchema,
|
||||
preset *api.PresetUpsertSchema,
|
||||
) *BaseAPI[indexDocument, returnType] {
|
||||
return &BaseAPI[indexDocument, returnType]{
|
||||
@ -48,11 +48,11 @@ func (b *BaseAPI[indexDocument, returnType]) Healthz(_ context.Context) error {
|
||||
}
|
||||
|
||||
// Indices returns a list of all configured index IDs
|
||||
func (b *BaseAPI[indexDocument, returnType]) Indices() ([]pkgtypesense.IndexID, error) {
|
||||
func (b *BaseAPI[indexDocument, returnType]) Indices() ([]pkgx.IndexID, error) {
|
||||
if len(b.collections) == 0 {
|
||||
return nil, errors.New("no collections configured")
|
||||
}
|
||||
indices := make([]pkgtypesense.IndexID, 0, len(b.collections))
|
||||
indices := make([]pkgx.IndexID, 0, len(b.collections))
|
||||
for index := range b.collections {
|
||||
indices = append(indices, index)
|
||||
}
|
||||
@ -84,19 +84,19 @@ func (b *BaseAPI[indexDocument, returnType]) Indices() ([]pkgtypesense.IndexID,
|
||||
// The system is considered valid if there is one alias for each collection and the collections
|
||||
// are correctly linked to their respective aliases.
|
||||
// The function sets the revisionID that is currently linked to the aliases internally.
|
||||
func (b *BaseAPI[indexDocument, returnType]) Initialize(ctx context.Context) (pkgtypesense.RevisionID, error) {
|
||||
b.l.Info("Initializing Typesense collections and aliases...")
|
||||
func (b *BaseAPI[indexDocument, returnType]) Initialize(ctx context.Context) (pkgx.RevisionID, error) {
|
||||
b.l.Info("initializing typesense collections and aliases...")
|
||||
|
||||
// Step 1: Check Typesense connection
|
||||
if _, err := b.client.Health(ctx, 5*time.Second); err != nil {
|
||||
b.l.Error("Typesense health check failed", zap.Error(err))
|
||||
b.l.Error("typesense health check failed", zap.Error(err))
|
||||
return "", err
|
||||
}
|
||||
|
||||
// Step 2: Retrieve existing aliases and collections
|
||||
aliases, err := b.client.Aliases().Retrieve(ctx)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to retrieve aliases", zap.Error(err))
|
||||
b.l.Error("failed to retrieve aliases", zap.Error(err))
|
||||
return "", err
|
||||
}
|
||||
|
||||
@ -106,12 +106,12 @@ func (b *BaseAPI[indexDocument, returnType]) Initialize(ctx context.Context) (pk
|
||||
}
|
||||
|
||||
// Step 3: Track latest revisions per alias
|
||||
latestRevisions := make(map[pkgtypesense.IndexID]pkgtypesense.RevisionID)
|
||||
aliasMappings := make(map[pkgtypesense.IndexID]string) // Tracks alias-to-collection mappings
|
||||
latestRevisions := make(map[pkgx.IndexID]pkgx.RevisionID)
|
||||
aliasMappings := make(map[pkgx.IndexID]string) // Tracks alias-to-collection mappings
|
||||
|
||||
for _, alias := range aliases {
|
||||
collectionName := alias.CollectionName
|
||||
indexID := pkgtypesense.IndexID(*alias.Name)
|
||||
indexID := pkgx.IndexID(*alias.Name)
|
||||
revisionID := extractRevisionID(collectionName, string(indexID))
|
||||
|
||||
// Ensure alias points to an existing collection
|
||||
@ -119,18 +119,18 @@ func (b *BaseAPI[indexDocument, returnType]) Initialize(ctx context.Context) (pk
|
||||
latestRevisions[indexID] = revisionID
|
||||
aliasMappings[indexID] = collectionName
|
||||
} else {
|
||||
b.l.Warn("Alias points to missing collection, resetting", zap.String("alias", string(indexID)))
|
||||
b.l.Warn("alias points to missing collection, resetting", zap.String("alias", string(indexID)))
|
||||
}
|
||||
}
|
||||
|
||||
// Step 4: Ensure all aliases are correctly mapped to collections and create a new revision
|
||||
newRevisionID := b.generateRevisionID()
|
||||
b.l.Info("Generated new revision", zap.String("revisionID", string(newRevisionID)))
|
||||
b.l.Info("generated new revision", zap.String("revisionID", string(newRevisionID)))
|
||||
|
||||
for indexID, schema := range b.collections {
|
||||
collectionName := formatCollectionName(indexID, newRevisionID)
|
||||
|
||||
b.l.Warn("Creating new collection & alias",
|
||||
b.l.Warn("creating new collection & alias",
|
||||
zap.String("index", string(indexID)),
|
||||
zap.String("new_collection", collectionName),
|
||||
)
|
||||
@ -153,24 +153,24 @@ func (b *BaseAPI[indexDocument, returnType]) Initialize(ctx context.Context) (pk
|
||||
if b.preset != nil {
|
||||
_, err := b.client.Presets().Upsert(ctx, defaultSearchPresetName, b.preset)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to upsert search preset", zap.Error(err))
|
||||
b.l.Error("failed to upsert search preset", zap.Error(err))
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
|
||||
b.l.Info("Initialization completed", zap.String("revisionID", string(b.revisionID)))
|
||||
b.l.Info("initialization completed", zap.String("revisionID", string(b.revisionID)))
|
||||
|
||||
return b.revisionID, nil
|
||||
}
|
||||
|
||||
func (b *BaseAPI[indexDocument, returnType]) UpsertDocuments(
|
||||
ctx context.Context,
|
||||
revisionID pkgtypesense.RevisionID,
|
||||
indexID pkgtypesense.IndexID,
|
||||
revisionID pkgx.RevisionID,
|
||||
indexID pkgx.IndexID,
|
||||
documents []*indexDocument,
|
||||
) error {
|
||||
if len(documents) == 0 {
|
||||
b.l.Warn("No documents provided for upsert", zap.String("index", string(indexID)))
|
||||
b.l.Warn("no documents provided for upsert", zap.String("index", string(indexID)))
|
||||
return nil
|
||||
}
|
||||
|
||||
@ -190,7 +190,7 @@ func (b *BaseAPI[indexDocument, returnType]) UpsertDocuments(
|
||||
|
||||
importResults, err := b.client.Collection(collectionName).Documents().Import(ctx, docInterfaces, params)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to bulk upsert documents", zap.String("collection", collectionName), zap.Error(err))
|
||||
b.l.Error("failed to bulk upsert documents", zap.String("collection", collectionName), zap.Error(err))
|
||||
return err
|
||||
}
|
||||
|
||||
@ -201,14 +201,14 @@ func (b *BaseAPI[indexDocument, returnType]) UpsertDocuments(
|
||||
successCount++
|
||||
} else {
|
||||
failureCount++
|
||||
b.l.Warn("Document failed to upsert",
|
||||
b.l.Warn("document failed to upsert",
|
||||
zap.String("collection", collectionName),
|
||||
zap.String("error", result.Error),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
b.l.Info("Bulk upsert completed",
|
||||
b.l.Info("bulk upsert completed",
|
||||
zap.String("collection", collectionName),
|
||||
zap.Int("successful_documents", successCount),
|
||||
zap.Int("failed_documents", failureCount),
|
||||
@ -220,7 +220,7 @@ func (b *BaseAPI[indexDocument, returnType]) UpsertDocuments(
|
||||
// it will update the aliases to point to the new revision
|
||||
// additionally it will remove all old collections that are not linked to an alias
|
||||
// keeping only the latest revision and the one before
|
||||
func (b *BaseAPI[indexDocument, returnType]) CommitRevision(ctx context.Context, revisionID pkgtypesense.RevisionID) error {
|
||||
func (b *BaseAPI[indexDocument, returnType]) CommitRevision(ctx context.Context, revisionID pkgx.RevisionID) error {
|
||||
for indexID := range b.collections {
|
||||
alias := string(indexID)
|
||||
newCollectionName := formatCollectionName(indexID, revisionID)
|
||||
@ -231,15 +231,15 @@ func (b *BaseAPI[indexDocument, returnType]) CommitRevision(ctx context.Context,
|
||||
CollectionName: newCollectionName,
|
||||
})
|
||||
if err != nil {
|
||||
b.l.Error("Failed to update alias", zap.String("alias", alias), zap.Error(err))
|
||||
b.l.Error("failed to update alias", zap.String("alias", alias), zap.Error(err))
|
||||
return err
|
||||
}
|
||||
b.l.Info("Updated alias", zap.String("alias", alias), zap.String("collection", newCollectionName))
|
||||
b.l.Info("updated alias", zap.String("alias", alias), zap.String("collection", newCollectionName))
|
||||
|
||||
// Step 2: Clean up old collections (keep only the last two)
|
||||
err = b.pruneOldCollections(ctx, alias, newCollectionName)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to clean up old collections", zap.String("alias", alias), zap.Error(err))
|
||||
b.l.Error("failed to clean up old collections", zap.String("alias", alias), zap.Error(err))
|
||||
}
|
||||
}
|
||||
|
||||
@ -247,18 +247,18 @@ func (b *BaseAPI[indexDocument, returnType]) CommitRevision(ctx context.Context,
|
||||
}
|
||||
|
||||
// RevertRevision will remove the collections created for the given revisionID
|
||||
func (b *BaseAPI[indexDocument, returnType]) RevertRevision(ctx context.Context, revisionID pkgtypesense.RevisionID) error {
|
||||
func (b *BaseAPI[indexDocument, returnType]) RevertRevision(ctx context.Context, revisionID pkgx.RevisionID) error {
|
||||
for indexID := range b.collections {
|
||||
collectionName := formatCollectionName(indexID, revisionID)
|
||||
|
||||
// Step 1: Delete the collection safely
|
||||
_, err := b.client.Collection(collectionName).Delete(ctx)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to delete collection", zap.String("collection", collectionName), zap.Error(err))
|
||||
b.l.Error("failed to delete collection", zap.String("collection", collectionName), zap.Error(err))
|
||||
return err
|
||||
}
|
||||
|
||||
b.l.Info("Reverted and deleted collection", zap.String("collection", collectionName))
|
||||
b.l.Info("reverted and deleted collection", zap.String("collection", collectionName))
|
||||
}
|
||||
|
||||
return nil
|
||||
@ -268,12 +268,12 @@ func (b *BaseAPI[indexDocument, returnType]) RevertRevision(ctx context.Context,
|
||||
// it will return the documents and the scores
|
||||
func (b *BaseAPI[indexDocument, returnType]) SimpleSearch(
|
||||
ctx context.Context,
|
||||
index pkgtypesense.IndexID,
|
||||
index pkgx.IndexID,
|
||||
q string,
|
||||
filterBy map[string][]string,
|
||||
page, perPage int,
|
||||
sortBy string,
|
||||
) ([]returnType, pkgtypesense.Scores, int, error) {
|
||||
) ([]returnType, pkgx.Scores, int, error) {
|
||||
// Call buildSearchParams but also set QueryBy explicitly
|
||||
parameters := buildSearchParams(q, filterBy, page, perPage, sortBy)
|
||||
parameters.QueryBy = pointer.String("title")
|
||||
@ -285,18 +285,18 @@ func (b *BaseAPI[indexDocument, returnType]) SimpleSearch(
|
||||
// it will return the documents, scores, and totalResults
|
||||
func (b *BaseAPI[indexDocument, returnType]) ExpertSearch(
|
||||
ctx context.Context,
|
||||
indexID pkgtypesense.IndexID,
|
||||
indexID pkgx.IndexID,
|
||||
parameters *api.SearchCollectionParams,
|
||||
) ([]returnType, pkgtypesense.Scores, int, error) {
|
||||
) ([]returnType, pkgx.Scores, int, error) {
|
||||
if parameters == nil {
|
||||
b.l.Error("Search parameters are nil")
|
||||
b.l.Error("search parameters are nil")
|
||||
return nil, nil, 0, errors.New("search parameters cannot be nil")
|
||||
}
|
||||
|
||||
collectionName := string(indexID) // digital-bks-at-de
|
||||
searchResponse, err := b.client.Collection(collectionName).Documents().Search(ctx, parameters)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to perform search", zap.String("index", collectionName), zap.Error(err))
|
||||
b.l.Error("failed to perform search", zap.String("index", collectionName), zap.Error(err))
|
||||
return nil, nil, 0, err
|
||||
}
|
||||
// Extract totalResults from the search response
|
||||
@ -304,16 +304,16 @@ func (b *BaseAPI[indexDocument, returnType]) ExpertSearch(
|
||||
|
||||
// Ensure Hits is not empty before proceeding
|
||||
if searchResponse.Hits == nil || len(*searchResponse.Hits) == 0 {
|
||||
b.l.Warn("Search response contains no hits", zap.String("index", collectionName))
|
||||
b.l.Warn("search response contains no hits", zap.String("index", collectionName))
|
||||
return nil, nil, totalResults, nil
|
||||
}
|
||||
|
||||
results := make([]returnType, len(*searchResponse.Hits))
|
||||
scores := make(pkgtypesense.Scores)
|
||||
scores := make(pkgx.Scores)
|
||||
|
||||
for i, hit := range *searchResponse.Hits {
|
||||
if hit.Document == nil {
|
||||
b.l.Warn("Hit document is nil", zap.String("index", collectionName))
|
||||
b.l.Warn("hit document is nil", zap.String("index", collectionName))
|
||||
continue
|
||||
}
|
||||
|
||||
@ -322,15 +322,19 @@ func (b *BaseAPI[indexDocument, returnType]) ExpertSearch(
|
||||
// Extract document ID safely
|
||||
docID, ok := docMap["id"].(string)
|
||||
if !ok {
|
||||
b.l.Warn("Missing or invalid document ID in search result")
|
||||
b.l.Warn("missing or invalid document ID in search result")
|
||||
continue
|
||||
}
|
||||
|
||||
// Convert hit to JSON and then unmarshal into returnType
|
||||
hitJSON, _ := json.Marshal(docMap)
|
||||
hitJSON, err := json.Marshal(docMap)
|
||||
if err != nil {
|
||||
b.l.Warn("failed to marshal document to JSON", zap.String("index", collectionName), zap.Error(err))
|
||||
continue
|
||||
}
|
||||
var doc returnType
|
||||
if err := json.Unmarshal(hitJSON, &doc); err != nil {
|
||||
b.l.Warn("Failed to unmarshal search result", zap.String("index", collectionName), zap.Error(err))
|
||||
b.l.Warn("failed to unmarshal JSON into returnType", zap.String("index", collectionName), zap.Error(err))
|
||||
continue
|
||||
}
|
||||
|
||||
@ -340,17 +344,17 @@ func (b *BaseAPI[indexDocument, returnType]) ExpertSearch(
|
||||
if score, err := strconv.Atoi(*hit.TextMatchInfo.Score); err == nil {
|
||||
index = score
|
||||
} else {
|
||||
b.l.Warn("Invalid score value", zap.String("score", *hit.TextMatchInfo.Score), zap.Error(err))
|
||||
b.l.Warn("invalid score value", zap.String("score", *hit.TextMatchInfo.Score), zap.Error(err))
|
||||
}
|
||||
}
|
||||
|
||||
scores[pkgtypesense.DocumentID(docID)] = pkgtypesense.Score{
|
||||
ID: pkgtypesense.DocumentID(docID),
|
||||
scores[pkgx.DocumentID(docID)] = pkgx.Score{
|
||||
ID: pkgx.DocumentID(docID),
|
||||
Index: index,
|
||||
}
|
||||
}
|
||||
|
||||
b.l.Info("Search completed",
|
||||
b.l.Info("search completed",
|
||||
zap.String("index", collectionName),
|
||||
zap.Int("results_count", len(results)),
|
||||
zap.Int("total_results", totalResults),
|
||||
|
||||
@ -7,7 +7,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
pkgtypesense "github.com/foomo/typesense/pkg"
|
||||
pkgx "github.com/foomo/typesense/pkg"
|
||||
"github.com/typesense/typesense-go/v3/typesense/api"
|
||||
"github.com/typesense/typesense-go/v3/typesense/api/pointer"
|
||||
"go.uber.org/zap"
|
||||
@ -59,15 +59,15 @@ func formatFilterQuery(filterBy map[string][]string) string {
|
||||
return strings.Join(filterClauses, " && ")
|
||||
}
|
||||
|
||||
func (b *BaseAPI[indexDocument, returnType]) generateRevisionID() pkgtypesense.RevisionID {
|
||||
return pkgtypesense.RevisionID(time.Now().Format("2006-01-02-15-04")) // "YYYY-MM-DD-HH-MM"
|
||||
func (b *BaseAPI[indexDocument, returnType]) generateRevisionID() pkgx.RevisionID {
|
||||
return pkgx.RevisionID(time.Now().Format("2006-01-02-15-04")) // "YYYY-MM-DD-HH-MM"
|
||||
}
|
||||
|
||||
func formatCollectionName(indexID pkgtypesense.IndexID, revisionID pkgtypesense.RevisionID) string {
|
||||
func formatCollectionName(indexID pkgx.IndexID, revisionID pkgx.RevisionID) string {
|
||||
return fmt.Sprintf("%s-%s", indexID, revisionID)
|
||||
}
|
||||
|
||||
func extractRevisionID(collectionName, name string) pkgtypesense.RevisionID {
|
||||
func extractRevisionID(collectionName, name string) pkgx.RevisionID {
|
||||
if !strings.HasPrefix(collectionName, name+"-") {
|
||||
return ""
|
||||
}
|
||||
@ -79,16 +79,16 @@ func extractRevisionID(collectionName, name string) pkgtypesense.RevisionID {
|
||||
return ""
|
||||
}
|
||||
|
||||
return pkgtypesense.RevisionID(revisionID)
|
||||
return pkgx.RevisionID(revisionID)
|
||||
}
|
||||
|
||||
// ensureAliasMapping ensures an alias correctly points to the specified collection.
|
||||
func (b *BaseAPI[indexDocument, returnType]) ensureAliasMapping(ctx context.Context, indexID pkgtypesense.IndexID, collectionName string) error {
|
||||
func (b *BaseAPI[indexDocument, returnType]) ensureAliasMapping(ctx context.Context, indexID pkgx.IndexID, collectionName string) error {
|
||||
_, err := b.client.Aliases().Upsert(ctx, string(indexID), &api.CollectionAliasSchema{
|
||||
CollectionName: collectionName,
|
||||
})
|
||||
if err != nil {
|
||||
b.l.Error("Failed to upsert alias",
|
||||
b.l.Error("failed to upsert alias",
|
||||
zap.String("alias", string(indexID)),
|
||||
zap.String("collection", collectionName),
|
||||
zap.Error(err),
|
||||
@ -101,7 +101,7 @@ func (b *BaseAPI[indexDocument, returnType]) pruneOldCollections(ctx context.Con
|
||||
// Step 1: Retrieve all collections
|
||||
collections, err := b.client.Collections().Retrieve(ctx)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to retrieve collections", zap.Error(err))
|
||||
b.l.Error("failed to retrieve collections", zap.Error(err))
|
||||
return err
|
||||
}
|
||||
|
||||
@ -123,9 +123,9 @@ func (b *BaseAPI[indexDocument, returnType]) pruneOldCollections(ctx context.Con
|
||||
for _, col := range toDelete {
|
||||
_, err := b.client.Collection(col).Delete(ctx)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to delete collection", zap.String("collection", col), zap.Error(err))
|
||||
b.l.Error("failed to delete collection", zap.String("collection", col), zap.Error(err))
|
||||
} else {
|
||||
b.l.Info("Deleted old collection", zap.String("collection", col))
|
||||
b.l.Info("deleted old collection", zap.String("collection", col))
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -137,7 +137,7 @@ func (b *BaseAPI[indexDocument, returnType]) pruneOldCollections(ctx context.Con
|
||||
func (b *BaseAPI[indexDocument, returnType]) fetchExistingCollections(ctx context.Context) (map[string]bool, error) {
|
||||
collections, err := b.client.Collections().Retrieve(ctx)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to retrieve collections", zap.Error(err))
|
||||
b.l.Error("failed to retrieve collections", zap.Error(err))
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@ -158,7 +158,7 @@ func (b *BaseAPI[indexDocument, returnType]) createCollectionIfNotExists(ctx con
|
||||
}
|
||||
|
||||
if existingCollections[collectionName] {
|
||||
b.l.Info("Collection already exists, skipping creation", zap.String("collection", collectionName))
|
||||
b.l.Info("collection already exists, skipping creation", zap.String("collection", collectionName))
|
||||
return nil
|
||||
}
|
||||
|
||||
@ -166,10 +166,10 @@ func (b *BaseAPI[indexDocument, returnType]) createCollectionIfNotExists(ctx con
|
||||
schema.Name = collectionName
|
||||
_, err = b.client.Collections().Create(ctx, schema)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to create collection", zap.String("collection", collectionName), zap.Error(err))
|
||||
b.l.Error("failed to create collection", zap.String("collection", collectionName), zap.Error(err))
|
||||
return err
|
||||
}
|
||||
|
||||
b.l.Info("Created new collection", zap.String("collection", collectionName))
|
||||
b.l.Info("created new collection", zap.String("collection", collectionName))
|
||||
return nil
|
||||
}
|
||||
|
||||
@ -5,23 +5,23 @@ import (
|
||||
"fmt"
|
||||
"slices"
|
||||
|
||||
"github.com/foomo/contentserver/client"
|
||||
contentserverclient "github.com/foomo/contentserver/client"
|
||||
"github.com/foomo/contentserver/content"
|
||||
typesense "github.com/foomo/typesense/pkg"
|
||||
pkgx "github.com/foomo/typesense/pkg"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
type ContentServer[indexDocument any] struct {
|
||||
l *zap.Logger
|
||||
contentserverClient *client.Client
|
||||
documentProviderFuncs map[typesense.DocumentType]typesense.DocumentProviderFunc[indexDocument]
|
||||
contentserverClient *contentserverclient.Client
|
||||
documentProviderFuncs map[pkgx.DocumentType]pkgx.DocumentProviderFunc[indexDocument]
|
||||
supportedMimeTypes []string
|
||||
}
|
||||
|
||||
func NewContentServer[indexDocument any](
|
||||
l *zap.Logger,
|
||||
client *client.Client,
|
||||
documentProviderFuncs map[typesense.DocumentType]typesense.DocumentProviderFunc[indexDocument],
|
||||
client *contentserverclient.Client,
|
||||
documentProviderFuncs map[pkgx.DocumentType]pkgx.DocumentProviderFunc[indexDocument],
|
||||
supportedMimeTypes []string,
|
||||
) *ContentServer[indexDocument] {
|
||||
return &ContentServer[indexDocument]{
|
||||
@ -34,7 +34,7 @@ func NewContentServer[indexDocument any](
|
||||
|
||||
func (c ContentServer[indexDocument]) Provide(
|
||||
ctx context.Context,
|
||||
indexID typesense.IndexID,
|
||||
indexID pkgx.IndexID,
|
||||
) ([]*indexDocument, error) {
|
||||
documentInfos, err := c.getDocumentIDsByIndexID(ctx, indexID)
|
||||
if err != nil {
|
||||
@ -71,7 +71,7 @@ func (c ContentServer[indexDocument]) Provide(
|
||||
|
||||
func (c ContentServer[indexDocument]) ProvidePaged(
|
||||
ctx context.Context,
|
||||
indexID typesense.IndexID,
|
||||
indexID pkgx.IndexID,
|
||||
offset int,
|
||||
) ([]*indexDocument, int, error) {
|
||||
panic("implement me")
|
||||
@ -79,8 +79,8 @@ func (c ContentServer[indexDocument]) ProvidePaged(
|
||||
|
||||
func (c ContentServer[indexDocument]) getDocumentIDsByIndexID(
|
||||
ctx context.Context,
|
||||
indexID typesense.IndexID,
|
||||
) ([]typesense.DocumentInfo, error) {
|
||||
indexID pkgx.IndexID,
|
||||
) ([]pkgx.DocumentInfo, error) {
|
||||
// get the contentserver dimension defined by indexID
|
||||
// create the list of document infos
|
||||
repo, err := c.contentserverClient.GetRepo(ctx)
|
||||
@ -93,12 +93,12 @@ func (c ContentServer[indexDocument]) getDocumentIDsByIndexID(
|
||||
}
|
||||
|
||||
nodeMap := createFlatRepoNodeMap(rootRepoNode, map[string]*content.RepoNode{})
|
||||
documentInfos := make([]typesense.DocumentInfo, 0, len(nodeMap))
|
||||
documentInfos := make([]pkgx.DocumentInfo, 0, len(nodeMap))
|
||||
for _, repoNode := range nodeMap {
|
||||
if slices.Contains(c.supportedMimeTypes, repoNode.MimeType) {
|
||||
documentInfos = append(documentInfos, typesense.DocumentInfo{
|
||||
DocumentType: typesense.DocumentType(repoNode.MimeType),
|
||||
DocumentID: typesense.DocumentID(repoNode.ID),
|
||||
documentInfos = append(documentInfos, pkgx.DocumentInfo{
|
||||
DocumentType: pkgx.DocumentType(repoNode.MimeType),
|
||||
DocumentID: pkgx.DocumentID(repoNode.ID),
|
||||
})
|
||||
}
|
||||
}
|
||||
@ -122,9 +122,9 @@ func createFlatRepoNodeMap(node *content.RepoNode, nodeMap map[string]*content.R
|
||||
|
||||
func (c ContentServer[indexDocument]) fetchURLsByDocumentIDs(
|
||||
ctx context.Context,
|
||||
indexID typesense.IndexID,
|
||||
documentInfos []typesense.DocumentInfo,
|
||||
) (map[typesense.DocumentID]string, error) {
|
||||
indexID pkgx.IndexID,
|
||||
documentInfos []pkgx.DocumentInfo,
|
||||
) (map[pkgx.DocumentID]string, error) {
|
||||
ids := make([]string, len(documentInfos))
|
||||
|
||||
for i, documentInfo := range documentInfos {
|
||||
@ -140,10 +140,10 @@ func (c ContentServer[indexDocument]) fetchURLsByDocumentIDs(
|
||||
return convertMapStringToDocumentID(uriMap), nil
|
||||
}
|
||||
|
||||
func convertMapStringToDocumentID(input map[string]string) map[typesense.DocumentID]string {
|
||||
output := make(map[typesense.DocumentID]string, len(input))
|
||||
func convertMapStringToDocumentID(input map[string]string) map[pkgx.DocumentID]string {
|
||||
output := make(map[pkgx.DocumentID]string, len(input))
|
||||
for key, value := range input {
|
||||
output[typesense.DocumentID(key)] = value
|
||||
output[pkgx.DocumentID(key)] = value
|
||||
}
|
||||
return output
|
||||
}
|
||||
|
||||
@ -3,20 +3,20 @@ package typesenseindexing
|
||||
import (
|
||||
"context"
|
||||
|
||||
typesense "github.com/foomo/typesense/pkg"
|
||||
pkgx "github.com/foomo/typesense/pkg"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
type BaseIndexer[indexDocument any, returnType any] struct {
|
||||
l *zap.Logger
|
||||
typesenseAPI typesense.API[indexDocument, returnType]
|
||||
documentProvider typesense.DocumentProvider[indexDocument]
|
||||
typesenseAPI pkgx.API[indexDocument, returnType]
|
||||
documentProvider pkgx.DocumentProvider[indexDocument]
|
||||
}
|
||||
|
||||
func NewBaseIndexer[indexDocument any, returnType any](
|
||||
l *zap.Logger,
|
||||
typesenseAPI typesense.API[indexDocument, returnType],
|
||||
documentProvider typesense.DocumentProvider[indexDocument],
|
||||
typesenseAPI pkgx.API[indexDocument, returnType],
|
||||
documentProvider pkgx.DocumentProvider[indexDocument],
|
||||
) *BaseIndexer[indexDocument, returnType] {
|
||||
return &BaseIndexer[indexDocument, returnType]{
|
||||
l: l,
|
||||
@ -33,14 +33,14 @@ func (b *BaseIndexer[indexDocument, returnType]) Run(ctx context.Context) error
|
||||
// Step 1: Ensure Typesense is initialized
|
||||
revisionID, err := b.typesenseAPI.Initialize(ctx)
|
||||
if err != nil || revisionID == "" {
|
||||
b.l.Error("Failed to initialize Typesense", zap.Error(err))
|
||||
b.l.Error("failed to initialize typesense", zap.Error(err))
|
||||
return err
|
||||
}
|
||||
|
||||
// Step 2: Retrieve all configured indices
|
||||
indices, err := b.typesenseAPI.Indices()
|
||||
if err != nil {
|
||||
b.l.Error("Failed to retrieve indices from Typesense", zap.Error(err))
|
||||
b.l.Error("failed to retrieve indices from typesense", zap.Error(err))
|
||||
return err
|
||||
}
|
||||
|
||||
@ -52,7 +52,7 @@ func (b *BaseIndexer[indexDocument, returnType]) Run(ctx context.Context) error
|
||||
// Fetch documents from the provider
|
||||
documents, err := b.documentProvider.Provide(ctx, indexID)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to fetch documents", zap.String("index", string(indexID)), zap.Error(err))
|
||||
b.l.Error("failed to fetch documents", zap.String("index", string(indexID)), zap.Error(err))
|
||||
tainted = true
|
||||
continue
|
||||
}
|
||||
@ -60,7 +60,7 @@ func (b *BaseIndexer[indexDocument, returnType]) Run(ctx context.Context) error
|
||||
err = b.typesenseAPI.UpsertDocuments(ctx, revisionID, indexID, documents)
|
||||
if err != nil {
|
||||
b.l.Error(
|
||||
"Failed to upsert documents",
|
||||
"failed to upsert documents",
|
||||
zap.String("index", string(indexID)),
|
||||
zap.String("revision", string(revisionID)),
|
||||
zap.Int("documents", len(documents)),
|
||||
@ -71,7 +71,7 @@ func (b *BaseIndexer[indexDocument, returnType]) Run(ctx context.Context) error
|
||||
}
|
||||
|
||||
indexedDocuments += len(documents)
|
||||
b.l.Info("Successfully upserted documents",
|
||||
b.l.Info("successfully upserted documents",
|
||||
zap.String("index", string(indexID)),
|
||||
zap.Int("count", len(documents)),
|
||||
)
|
||||
@ -82,20 +82,20 @@ func (b *BaseIndexer[indexDocument, returnType]) Run(ctx context.Context) error
|
||||
// No errors encountered, commit the revision
|
||||
err = b.typesenseAPI.CommitRevision(ctx, revisionID)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to commit revision", zap.String("revision", string(revisionID)), zap.Error(err))
|
||||
b.l.Error("failed to commit revision", zap.String("revision", string(revisionID)), zap.Error(err))
|
||||
return err
|
||||
}
|
||||
b.l.Info("Successfully committed revision", zap.String("revision", string(revisionID)))
|
||||
b.l.Info("successfully committed revision", zap.String("revision", string(revisionID)))
|
||||
} else {
|
||||
// If errors occurred, revert the revision
|
||||
b.l.Warn("Errors detected during upsert, reverting revision", zap.String("revision", string(revisionID)))
|
||||
b.l.Warn("errors detected during upsert, reverting revision", zap.String("revision", string(revisionID)))
|
||||
|
||||
err = b.typesenseAPI.RevertRevision(ctx, revisionID)
|
||||
if err != nil {
|
||||
b.l.Error("Failed to revert revision", zap.String("revision", string(revisionID)), zap.Error(err))
|
||||
b.l.Error("failed to revert revision", zap.String("revision", string(revisionID)), zap.Error(err))
|
||||
return err
|
||||
}
|
||||
b.l.Info("Successfully reverted revision", zap.String("revision", string(revisionID)))
|
||||
b.l.Info("successfully reverted revision", zap.String("revision", string(revisionID)))
|
||||
}
|
||||
|
||||
return nil
|
||||
|
||||
Loading…
Reference in New Issue
Block a user