10 Star 42 Fork 13

unionj-cloud / go-doudou

Create your Gitee Account
Explore and code with more than 6 million developers,Free private repositories !:)
Sign up
Clone or Download
Cancel
Notice: Creating folder will generate an empty file .keep, because not support in Git
Loading...
README.md

go-doudou

Mentioned in Awesome Go GoDoc Build Status Go codecov Go Report Card Release Goproxy.cn License: MIT FOSSA Status Slack

EN 中文
go-doudou(doudou pronounce /dəudəu/)is a gossip protocol and OpenAPI 3.0 spec based decentralized microservice framework. It supports monolith service application as well. Currently, it supports restful service only.

TOC

Philosophy

  • Design First: We encourage designing your apis at the first place.
  • Contract: We use OpenAPI 3.0 spec as a contract between server and client to reduce the communication cost between different dev teams and speed up development.
  • Decentralization: We use gossip protocol to do service register and discovery to build a robust, scalable and decentralized service cluster. Thanks the awesome library memberlist by hashicorp.

Features

  • Low-code: design service interface to generate main function, routes, http handlers, mock service implementation, http client, OpenAPI 3.0 json spec and more.
  • Support DNS address for service register and discovery
  • Support monolith and microservices architecture
  • Built-in client load balancing: currently only round-robin
  • Built-in graceful shutdown
  • Built-in live reloading by watching go files(not support windows)
  • Built-in service apis documentation UI
  • Built-in service registry UI
  • Built-in prometheus middlewares: http_requests_total, response_status and http_response_time_seconds
  • Built-in docker and k8s deployment support: dockerfile, deployment kind yaml file and statefulset kind yaml file
  • Easy to learn, simple to use

Overview

Overview

Recommend Architecture

Recommend Architecture

Golang Version Support

  • go 1.13, 1.14, 1.15 with GO111MODULE=on
  • go 1.16+
  • < go 1.13: not test, maybe support

Install

go get -v github.com/unionj-cloud/go-doudou@v0.7.7

Usage

➜  ~ go-doudou -h                            
WARN[0000] Error loading .env file: open /Users/.env: no such file or directory 
go-doudou works like a scaffolding tool but more than that. 
it lets api providers design their apis and help them code less. 
it generates openapi 3.0 spec json document for frontend developers or other api consumers to understand what apis there, 
consumers can import it into postman to debug and test, or upload it into some code generators to download client sdk.
it provides some useful components and middleware for constructing microservice cluster like service register and discovering, 
load balancing and so on. it just begins, more features will come out soon.

Usage:
  go-doudou [flags]
  go-doudou [command]

Available Commands:
  ddl         migration tool between database table structure and golang struct
  help        Help about any command
  name        bulk add or update struct fields json tag
  svc         generate or update service

Flags:
  -h, --help      help for go-doudou
  -v, --version   version for go-doudou

Use "go-doudou [command] --help" for more information about a command.

Hello World

Initialize project

➜  ~ go-doudou svc init helloworld
WARN[0000] Error loading .env file: open /Users/.env: no such file or directory 
1.16
helloworld

You can ignore the warning now.

➜  helloworld git:(master)ls -la -h
total 40
drwxr-xr-x   10 wubin1989  staff   320B  8 29 23:27 .
drwxr-xr-x+ 157 wubin1989  staff   4.9K  8 29 23:27 ..
-rw-r--r--    1 wubin1989  staff   2.0K  8 29 23:22 .env
drwxr-xr-x    5 wubin1989  staff   160B  8 29 23:26 .git
-rw-r--r--    1 wubin1989  staff   268B  8 29 23:22 .gitignore
drwxr-xr-x    6 wubin1989  staff   192B  8 29 23:27 .idea
-rw-r--r--    1 wubin1989  staff   707B  8 29 23:22 Dockerfile
-rw-r--r--    1 wubin1989  staff   442B  8 29 23:22 go.mod
-rw-r--r--    1 wubin1989  staff   253B  8 29 23:22 svc.go
drwxr-xr-x    3 wubin1989  staff    96B  8 29 23:22 vo
  • Dockerfile:build docker image

  • svc.go: design your rest apis by defining methods of Helloworld interface

  • vo folder:define structs as data structure in http request body and response body, and also as OpenAPI 3.0 schemas

  • .env: config file, go-doudou use it to load enviroment variables with GDD_ prefix

Define methods

Please read Must Know

package service

import (
	"context"
	"helloworld/vo"
)

type Helloworld interface {
	// You can define your service methods as your need. Below is an example.
	PageUsers(ctx context.Context, query vo.PageQuery) (code int, data vo.PageRet, err error)
}

Generate code

go-doudou svc http --handler -c go -o --doc
go mod tidy

Let's see what are generated.

➜  helloworld git:(master)ls -la -h
total 328
drwxr-xr-x   20 wubin1989  staff   640B  8 31 12:34 .
drwxr-xr-x+ 157 wubin1989  staff   4.9K  8 31 12:36 ..
-rw-r--r--    1 wubin1989  staff   2.0K  8 29 23:45 .env
drwxr-xr-x    5 wubin1989  staff   160B  8 31 12:36 .git
-rw-r--r--    1 wubin1989  staff   268B  8 29 23:22 .gitignore
drwxr-xr-x    7 wubin1989  staff   224B  8 31 12:33 .idea
-rw-r--r--    1 wubin1989  staff   707B  8 29 23:22 Dockerfile
-rwxr-xr-x    1 wubin1989  staff    13K  8 31 12:35 app.log
drwxr-xr-x    3 wubin1989  staff    96B  8 29 23:44 client
drwxr-xr-x    3 wubin1989  staff    96B  8 29 23:44 cmd
drwxr-xr-x    3 wubin1989  staff    96B  8 29 23:44 config
drwxr-xr-x    3 wubin1989  staff    96B  8 29 23:44 db
-rw-r--r--    1 wubin1989  staff   536B  8 31 12:35 go.mod
-rw-r--r--    1 wubin1989  staff   115K  8 31 12:35 go.sum
-rwxr-xr-x    1 wubin1989  staff   1.9K  8 31 12:34 helloworld_openapi3.go
-rwxr-xr-x    1 wubin1989  staff   1.8K  8 31 12:34 helloworld_openapi3.json
-rw-r--r--    1 wubin1989  staff   253B  8 29 23:22 svc.go
-rw-r--r--    1 wubin1989  staff   413B  8 29 23:44 svcimpl.go
drwxr-xr-x    3 wubin1989  staff    96B  8 29 23:44 transport
drwxr-xr-x    3 wubin1989  staff    96B  8 29 23:22 vo
  • helloworld_openapi3.json:OpenAPI 3.0 spec json documentation
  • helloworld_openapi3.go: assign OpenAPI 3.0 spec json string to a variable for serving online
  • client:golang http client based on resty
  • cmd:main.go file here
  • config:config loading related
  • db:function for connecting to database
  • svcimpl.go:write your business logic here
  • transport:http routes and handlers
  • .env:put configs here

Run

Set GDD_MEM_SEED empty in .env file because there is no seed address before run our service now.

➜  helloworld git:(master) ✗ go run cmd/main.go
INFO[2021-08-31 21:35:47] Node 192.168.2.20 joined, supplying helloworld service 
WARN[2021-08-31 21:35:47] No seed found                                
INFO[2021-08-31 21:35:47] Memberlist created. Local node is Node 192.168.2.20, providing helloworld service at http://192.168.2.20:6060, memberlist port 50324 
 _____                     _                    _
|  __ \                   | |                  | |
| |  \/  ___   ______   __| |  ___   _   _   __| |  ___   _   _
| | __  / _ \ |______| / _` | / _ \ | | | | / _` | / _ \ | | | |
| |_\ \| (_) |        | (_| || (_) || |_| || (_| || (_) || |_| |
 \____/ \___/          \__,_| \___/  \__,_| \__,_| \___/  \__,_|
INFO[2021-08-31 21:35:47] ================ Registered Routes ================ 
INFO[2021-08-31 21:35:47] +-------------+--------+-------------------------+ 
INFO[2021-08-31 21:35:47] |    NAME     | METHOD |         PATTERN         | 
INFO[2021-08-31 21:35:47] +-------------+--------+-------------------------+ 
INFO[2021-08-31 21:35:47] | PageUsers   | POST   | /page/users             | 
INFO[2021-08-31 21:35:47] | GetDoc      | GET    | /go-doudou/doc          | 
INFO[2021-08-31 21:35:47] | GetOpenAPI  | GET    | /go-doudou/openapi.json | 
INFO[2021-08-31 21:35:47] | Prometheus  | GET    | /go-doudou/prometheus   | 
INFO[2021-08-31 21:35:47] | GetRegistry | GET    | /go-doudou/registry     | 
INFO[2021-08-31 21:35:47] +-------------+--------+-------------------------+ 
INFO[2021-08-31 21:35:47] =================================================== 
INFO[2021-08-31 21:35:47] Started in 431.269µs                         
INFO[2021-08-31 21:35:47] Http server is listening on :6060

screenshot-doc screenshot-registry

Deployment

Build docker image and push to your repository
➜  helloworld git:(master) ✗ go-doudou svc push -r wubin1989
[+] Building 0.8s (13/13) FINISHED                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 37B                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                   0.0s
 => => transferring context: 2B                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/golang:1.13.4-alpine                                                             0.0s
 => [1/8] FROM docker.io/library/golang:1.13.4-alpine                                                                               0.0s
 => [internal] load build context                                                                                                   0.7s
 => => transferring context: 22.43MB                                                                                                0.6s
 => CACHED [2/8] WORKDIR /repo                                                                                                      0.0s
 => CACHED [3/8] ADD go.mod .                                                                                                       0.0s
 => CACHED [4/8] ADD go.sum .                                                                                                       0.0s
 => CACHED [5/8] ADD . ./                                                                                                           0.0s
 => CACHED [6/8] RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories                                   0.0s
 => CACHED [7/8] RUN apk add --no-cache bash tzdata                                                                                 0.0s
 => CACHED [8/8] RUN export GDD_VER=$(go list -mod=vendor -m -f '{{ .Version }}' github.com/unionj-cloud/go-doudou) && CGO_ENABLED  0.0s
 => exporting to image                                                                                                              0.0s
 => => exporting layers                                                                                                             0.0s
 => => writing image sha256:00365c58d0410d978aea462ec93323e20d879b15421e8eba29d8a17918660af8                                        0.0s
 => => naming to docker.io/library/helloworld                                                                                       0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
The push refers to repository [docker.io/wubin1989/helloworld]
d0a9599b03e1: Pushed 
c3055fdf1a79: Layer already exists 
1c265a7f4c3e: Layer already exists 
f567cf5a5cf1: Layer already exists 
0b4acd902364: Layer already exists 
bbf9670b59e9: Layer already exists 
fdd6fb6fca5b: Layer already exists 
a17f85ec7605: Layer already exists 
2895b872dff5: Layer already exists 
eed8c158e67f: Layer already exists 
2033402d2275: Layer already exists 
77cae8ab23bf: Layer already exists 
v20210831125525: digest: sha256:5f75f7b43708d0619555f9bccbf0347e8db65319b83c65251015982ca6d23370 size: 2829
time="2021-08-31 12:55:53" level=info msg="image wubin1989/helloworld:v20210831125525 has been pushed successfully\n"
time="2021-08-31 12:55:53" level=info msg="k8s yaml has been created/updated successfully. execute command 'go-doudou svc deploy' to deploy service helloworld to k8s cluster\n"

then you should see there are two yaml files generated

➜  helloworld git:(master) ✗ ll
total 328
-rw-r--r--  1 wubin1989  staff   707B  8 29 23:22 Dockerfile
-rwxr-xr-x  1 wubin1989  staff    15K  8 31 12:55 app.log
drwxr-xr-x  3 wubin1989  staff    96B  8 29 23:44 client
drwxr-xr-x  3 wubin1989  staff    96B  8 29 23:44 cmd
drwxr-xr-x  3 wubin1989  staff    96B  8 29 23:44 config
drwxr-xr-x  3 wubin1989  staff    96B  8 29 23:44 db
-rw-r--r--  1 wubin1989  staff   536B  8 31 12:35 go.mod
-rw-r--r--  1 wubin1989  staff   115K  8 31 12:35 go.sum
-rw-r--r--  1 wubin1989  staff   817B  8 31 12:55 helloworld_deployment.yaml
-rwxr-xr-x  1 wubin1989  staff   1.9K  8 31 12:34 helloworld_openapi3.go
-rwxr-xr-x  1 wubin1989  staff   1.8K  8 31 12:34 helloworld_openapi3.json
-rw-r--r--  1 wubin1989  staff   867B  8 31 12:55 helloworld_statefulset.yaml
-rw-r--r--  1 wubin1989  staff   253B  8 29 23:22 svc.go
-rw-r--r--  1 wubin1989  staff   413B  8 29 23:44 svcimpl.go
drwxr-xr-x  3 wubin1989  staff    96B  8 29 23:44 transport
drwxr-xr-x  6 wubin1989  staff   192B  8 31 12:55 vendor
drwxr-xr-x  3 wubin1989  staff    96B  8 29 23:22 vo
  • helloworld_deployment.yaml: k8s deploy file for stateless service, recommend for monolith architecture services
  • helloworld_statefulset.yaml: k8s deploy file for stateful service, recommend for microservices architecture services
Deploy
go-doudou svc deploy 
Shutdown
go-doudou svc shutdown

Must Know

There are some constraints or notable things when you define your methods as exposed apis for client in svc.go file.

  1. Only support GET, POST, PUT, DELETE http methods. If method name starts with one of Get/Post/Put/Delete, http method will be one of GET/POST/PUT/DELETE. If method name doesn't start with any of them, default http method is POST.
  2. First input parameter MUST be context.Context.
  3. Only support golang built-in types, map with string key, custom structs in vo package, corresponding slice and pointer types for input and output parameters. When go-doudou generate code and OpenAPI 3.0 spec, it will scan structs in vo package. If there is a struct from other package, the struct fields cannot be known by go-doudou.
  4. As special cases, it supports multipart.FileHeader for uploading file as input parameter, supports os.File for downloading file as output parameter.
  5. NOT support alias types as field of a struct.
  6. NOT support func, channel, interface and anonymous struct type as input and output parameter.
  7. When execute go-doudou svc http --handler , existing code in handlerimpl.go won't be overwritten. If you added methods in svc.go, new code will be appended to handlerimpl.go.
  8. When execute go-doudou svc http --handler , existing code in handler.go will be overwritten, so don't modify handler.go file.
  9. When execute go-doudou svc http, before generating one file other than handler.go and handlerimpl.go files, go-doudou will check if it exists first, if already exists, do nothing.

Service register & discovery

Go-doudou supports monolith and microservices architecture.

  • GDD_MODE=micro:microservices architecture
  • GDD_MODE=mono:monolith architecture

There is service register code in main.go file.

if ddconfig.GddMode.Load() == "micro" {
    node, err := registry.NewNode()
    if err != nil {
        logrus.Panicln(fmt.Sprintf("%+v", err))
    }
    logrus.Infof("Memberlist created. Local node is %s\n", node)
}

If dependent on other services, here is an example code below.

// service register
node, err := registry.NewNode()
if err != nil {
    logrus.Panicln(fmt.Sprintf("%+v", err))
}
logrus.Infof("%s joined cluster\n", node.String())

// call NewMemberlistServiceProvider to new a provider with name of the dependent service
usersvcProvider := ddhttp.NewMemberlistServiceProvider("usersvc", node)
// inject the provider into a client of the service
usersvcClient := client.NewUsersvc(client.WithProvider(usersvcProvider))

// inject the client into our service implementation instance
svc := service.NewOrdersvc(conf, conn, usersvcClient)

Client load balance

Currently only one round robin strategy, welcome pr:)

func (m *MemberlistServiceProvider) SelectServer() (string, error) {
	nodes, err := m.registry.Discover(m.name)
	if err != nil {
		return "", errors.Wrap(err, "SelectServer() fail")
	}
	next := int(atomic.AddUint64(&m.current, uint64(1)) % uint64(len(nodes)))
	m.current = uint64(next)
	selected := nodes[next]
	return selected.BaseUrl(), nil
}

Configuration

Go-doudou use .env file to load environment variables to configure behaviors.

Environment Variable Description Default Required
GDD_BANNER Whether output banner to stdout or not, possible values are on and off. off
GDD_BANNER_TEXT Go-doudou
GDD_LOG_LEVEL Possible values are panic, fatal, error, warn, warning, info, debug, trace info
GDD_LOG_PATH if GDD_LOG_PATH is not set, there is no output to disk.
GDD_GRACE_TIMEOUT Graceful shutdown timeout for http server 15s
GDD_WRITE_TIMEOUT Configure http.Server 15s
GDD_READ_TIMEOUT Configure http.Server 15s
GDD_IDLE_TIMEOUT Configure http.Server 60s
GDD_ROUTE_ROOT_PATH prefix GDD_ROUTE_ROOT_PATH to each of http api routes ""
GDD_SERVICE_NAME Service name that the node providing in the cluster. Yes
GDD_HOST Configure http.Server. Specifying host for the http server to listen on. ""
GDD_PORT Configure http.Server. Specifying port for the http server to listen on. ""
GDD_MODE Accept "mono" for monolith mode or "micro" for microservice mode
GDD_MANAGE_ENABLE Enable built-in api endpoints such as /go-doudou/doc, /go-doudou/openapi.json, /go-doudou/prometheus and /go-doudou/registry. Possible values are true and false. false
GDD_MANAGE_USER Http basic username for built-in api endpoints ""
GDD_MANAGE_PASS Http basic password for built-in api endpoints ""
GDD_MEM_SEED Seed address for join memberlist cluster. If empty or not set, this node will create a new cluster for other nodes to join. ""
GDD_MEM_NAME Only for dev and test use. Unique name of this node in cluster. if empty or not set, hostname will be used instead. ""
GDD_MEM_HOST Specify AdvertiseAddr attribute of memberlist config struct. if GDD_MEM_HOST starts with dot such as .seed-svc-headless.default.svc.cluster.local, it will be prefixed by hostname such as seed-2.seed-svc-headless.default.svc.cluster.local for supporting k8s stateful service. ""
GDD_MEM_PORT If empty or not set, an available port will be chosen randomly. Recommend specifying a port. ""
GDD_MEM_DEAD_TIMEOUT Dead node will be removed from node map if not received refute messages from it in GDD_MEM_DEAD_TIMEOUT second. 30
GDD_MEM_SYNC_INTERVAL Local node will synchronize states from other random node every GDD_MEM_SYNC_INTERVAL second. 5
GDD_MEM_RECLAIM_TIMEOUT Dead node will be replaced with new node with the same name but different full address in GDD_MEM_RECLAIM_TIMEOUT second 3

Example

Please check go-doudou-guide

Notable tools

name

Command line tool for generating json tag of struct field. Please check document .

ddl

DDL and dao layer generation command line tool based on jmoiron/sqlx. Please check document.

TODO

Please reference go-doudou kanban

Community

Welcome to contribute to go-doudou by forking it and submitting pr or issues. If you like go-doudou, please give it a star!

Slack invitation link: https://join.slack.com/t/go-doudou/shared_invite/zt-xzohc7ds-u7~aio6B8PELp5UtAdY~uw

Welcome to contact me from

License

FOSSA Status

Repository Comments ( 2 )

Sign in for post a comment

About

go-doudou(doudou发音/dəudəu/)是基于gossip协议做服务注册与发现,基于openapi 3.0规范做接口定义的go语言去中心化微服务敏捷开发框架。 expand collapse
Cancel

Releases (4)

All

go-doudou

Contributors

All

Activities

Load More
can not load any more
Go
1
https://git.oschina.net/unionj-cloud/go-doudou.git
git@git.oschina.net:unionj-cloud/go-doudou.git
unionj-cloud
go-doudou
go-doudou
main

Search