Skip to content

Commit 8860f4f

Browse files
bnkamaleshbnkamalesh
committed
- Nibbler package for micro batch processing
1 parent 10bf722 commit 8860f4f

File tree

8 files changed

+679
-0
lines changed

8 files changed

+679
-0
lines changed

‎.editorconfig

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
; https://editorconfig.org/
2+
3+
root = true
4+
5+
[*]
6+
insert_final_newline = true
7+
charset = utf-8
8+
trim_trailing_whitespace = true
9+
indent_style = space
10+
indent_size = 2
11+
12+
[{Makefile,go.mod,go.sum,*.go,.gitmodules}]
13+
indent_style = tab
14+
indent_size = 4
15+
16+
[*.md]
17+
indent_size = 4
18+
trim_trailing_whitespace = false
19+
20+
eclint_indent_style = unset

‎.github/workflows/go.yml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# This workflow will build a golang project
2+
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go
3+
4+
name: Go
5+
6+
on:
7+
push:
8+
branches: ["main"]
9+
pull_request:
10+
branches: ["main"]
11+
12+
jobs:
13+
build:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- name: Set up Go
19+
uses: actions/setup-go@v4
20+
with:
21+
go-version: "1.23"
22+
23+
- name: Build
24+
run: go build -v ./...
25+
26+
- name: Tests
27+
run: |
28+
go install github.com/mattn/goveralls@latest
29+
go test -race -covermode atomic -coverprofile=covprofile ./...
30+
31+
- name: Send coverage
32+
uses: shogo82148/actions-goveralls@v1
33+
with:
34+
path-to-profile: covprofile
35+
36+
- name: golangci-lint
37+
uses: golangci/golangci-lint-action@v6
38+
with:
39+
version: v1.60

‎.gitignore

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
# Created by https://www.toptal.com/developers/gitignore/api/go,visualstudiocode,linux,windows,macos
2+
# Edit at https://www.toptal.com/developers/gitignore?templates=go,visualstudiocode,linux,windows,macos
3+
4+
### Go ###
5+
# If you prefer the allow list template instead of the deny list, see community template:
6+
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
7+
#
8+
# Binaries for programs and plugins
9+
*.exe
10+
*.exe~
11+
*.dll
12+
*.so
13+
*.dylib
14+
15+
# Test binary, built with `go test -c`
16+
*.test
17+
18+
# Output of the go coverage tool, specifically when used with LiteIDE
19+
*.out
20+
21+
# Dependency directories (remove the comment below to include it)
22+
# vendor/
23+
24+
# Go workspace file
25+
go.work
26+
27+
### Linux ###
28+
*~
29+
30+
# temporary files which can be created if a process still has a handle open of a deleted file
31+
.fuse_hidden*
32+
33+
# KDE directory preferences
34+
.directory
35+
36+
# Linux trash folder which might appear on any partition or disk
37+
.Trash-*
38+
39+
# .nfs files are created when an open file is removed but is still being accessed
40+
.nfs*
41+
42+
### macOS ###
43+
# General
44+
.DS_Store
45+
.AppleDouble
46+
.LSOverride
47+
48+
# Icon must end with two \r
49+
Icon
50+
51+
52+
# Thumbnails
53+
._*
54+
55+
# Files that might appear in the root of a volume
56+
.DocumentRevisions-V100
57+
.fseventsd
58+
.Spotlight-V100
59+
.TemporaryItems
60+
.Trashes
61+
.VolumeIcon.icns
62+
.com.apple.timemachine.donotpresent
63+
64+
# Directories potentially created on remote AFP share
65+
.AppleDB
66+
.AppleDesktop
67+
Network Trash Folder
68+
Temporary Items
69+
.apdisk
70+
71+
### macOS Patch ###
72+
# iCloud generated files
73+
*.icloud
74+
75+
### VisualStudioCode ###
76+
.vscode/*
77+
!.vscode/settings.json
78+
!.vscode/tasks.json
79+
!.vscode/launch.json
80+
!.vscode/extensions.json
81+
!.vscode/*.code-snippets
82+
83+
# Local History for Visual Studio Code
84+
.history/
85+
86+
# Built Visual Studio Code Extensions
87+
*.vsix
88+
89+
### VisualStudioCode Patch ###
90+
# Ignore all local history of files
91+
.history
92+
.ionide
93+
94+
### Windows ###
95+
# Windows thumbnail cache files
96+
Thumbs.db
97+
Thumbs.db:encryptable
98+
ehthumbs.db
99+
ehthumbs_vista.db
100+
101+
# Dump file
102+
*.stackdump
103+
104+
# Folder config file
105+
[Dd]esktop.ini
106+
107+
# Recycle Bin used on file shares
108+
$RECYCLE.BIN/
109+
110+
# Windows Installer files
111+
*.cab
112+
*.msi
113+
*.msix
114+
*.msm
115+
*.msp
116+
117+
# Windows shortcuts
118+
*.lnk
119+
120+
# End of https://www.toptal.com/developers/gitignore/api/go,visualstudiocode,linux,windows,macos

‎README.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
[![](https://github.com/naughtygopher/nibbler/actions/workflows/go.yml/badge.svg?branch=main)](https://github.com/naughtygopher/nibbler/actions)
2+
[![Go Reference](https://pkg.go.dev/badge/github.com/naughtygopher/nibbler.svg)](https://pkg.go.dev/github.com/naughtygopher/nibbler)
3+
[![Go Report Card](https://goreportcard.com/badge/github.com/naughtygopher/nibbler)](https://goreportcard.com/report/github.com/naughtygopher/nibbler)
4+
[![Coverage Status](https://coveralls.io/repos/github/naughtygopher/nibbler/badge.svg?branch=main)](https://coveralls.io/github/naughtygopher/nibbler?branch=main)
5+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/creativecreature/sturdyc/blob/master/LICENSE)
6+
7+
# Nibbler
8+
9+
Nibbler is a minimal package which helps you implement micro-batch processing.
10+
11+
## What is Micro-batch Processing?
12+
13+
Micro-batch processing is a way to handle data by breaking a big task into smaller pieces and processing them one by one. This method is useful in real-time data or streaming situations, wher ,the incoming data is split into "micro-batches" and processed quickly, rather than waiting to collect all data at once.
14+
15+
The same concept can also be extended to handle events processing. So, we have a queue subscriber, and instead of processing the events individually, we create micro batches and process them.
16+
17+
The processing of a single micro batch can be triggered in two ways, based on a time ticker or if the micro batch size is full. i.e. process a non empty batch if duration X has passed or if the batch size is full
18+
19+
## How to use nibbler?
20+
21+
```golang
22+
23+
```

‎go.mod

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
module github.com/naughtygopher/nibbler
2+
3+
go 1.23.1
4+
5+
require github.com/stretchr/testify v1.9.0
6+
7+
require (
8+
github.com/davecgh/go-spew v1.1.1 // indirect
9+
github.com/pmezard/go-difflib v1.0.0 // indirect
10+
gopkg.in/yaml.v3 v3.0.1 // indirect
11+
)

‎go.sum

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
2+
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
3+
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
4+
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
5+
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
6+
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
7+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
8+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
9+
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
10+
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

0 commit comments

Comments
 (0)