kubernetes部署skywalking集群包括Java服务接入

1 概述：

1.1 环境

版本信息如下：
`
a、操作系统：centos 7.9

a、skywalking版本：v9.0.1

c、kubernetes版本：v1.22.0

d、es版本：6.8.6

e、helm版本： helm3.8

1.2 skywalking概述

1.2.1 skywalking是什么
SkyWalking是一个开源的APM系统，为云原生分布式系统提供监控、链路追踪、诊断能力，支持集成多种编程语言的应用（java、php、go、lua等），也能和服务网格进行集成。除了支持代码侵入方式的集成，一个主要亮点也支持零代码入侵的集成（零代码侵入是和具体的编程语言相关的），是利用java agent的特性在jvm级别修改了运行时的程序，因此程序员在代码编辑期间不需要修改业务代码也能达到埋点的效果。后端存储支持es、mysql、tidb等多种数据库。

架构图如下：

1.2.1 skywalking的java代理的使用

1）方式1：命令行方式

java \
-javaagent:/root/skywalking/agent/skywalking-agent.jar \
-Dskywalking.agent.service_name=app1 \
-Dskywalking.collector.backend_service=localhost:11800 \
-jar myapp.jar

2）方式2：环境变量方式

export SW_AGENT_COLLECTOR_BACKEND_SERVICES=10.0.0.1:11800,10.0.0.2:11800
export SW_AGENT_NAME=demo1
export JAVA_OPTS=-javaagent:/root/skywalking/agent/skywalking-agent.jar

java \
$JAVA_OPTS \
-jar myapp.jar

2 部署前置条件：

具备一个k8s集群：

3 部署：

3.1 部署es集群

cat > elasticsearch-deployment.yaml < EOF
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: elastic
spec:
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  serviceName: elasticsearch
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: elasticsearch
    spec:
      containers:
      - env:
        - name: cluster.name
          value: k8s-logs
        - name: node.name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: discovery.zen.ping.unicast.hosts
          value: elasticsearch-0.elasticsearch,elasticsearch-1.elasticsearch,elasticsearch-2.elasticsearch
        - name: discovery.zen.minimum_master_nodes
          value: "2"
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        image: docker.elastic.co/elasticsearch/elasticsearch:6.8.6
        imagePullPolicy: Always
        name: elasticsearch
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        resources:
          limits:
            cpu: "1"
          requests:
            cpu: 100m
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: elasticsearch-data-pvc
      initContainers:
      - command:
        - sh
        - -c
        - chown -R 1000:1000 /usr/share/elasticsearch/data
        image: busybox
        imagePullPolicy: Always
        name: fix-permissions
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name:  elasticsearch-data-pvc
      - command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        image: busybox
        imagePullPolicy: Always
        name: increase-vm-max-map
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - command:
        - sh
        - -c
        - ulimit -n 65536
        image: busybox
        imagePullPolicy: Always
        name: increase-fd-ulimit
        resources: {}
        securityContext:
          privileged: true
     #  volumes:
  volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data-pvc # 这里不要修改，进阶用法参考 ECK 官方文档
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi # 配置默认大小，allowVolumeExpansion为true后续可以扩展
        storageClassName: elasticsearch-nfs-sc
      #- emptyDir: {}
      #  name: data
      #- name: data
      #  persistentVolumeClaim:
      #    claimName: elasticsearch-data-pvc

---
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: elastic
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

---
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-logging
  namespace: elastic
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  ports:
    - port: 9200
      name: external
      
EOF                                                                                                                        ```                    
```shell
cat > elasticsearch-data-sc.yaml < EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: elasticsearch-nfs-sc
provisioner: fuseim.pri/ifs
EOF

cat >  elasticsearch-pvc.yaml
 < EOF
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: elasticsearch-data-pvc
  namespace: elastic
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: elasticsearch-nfs-sc
status:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 100Gi

EOF

es 集群地址：主机IP+port

3.2 部署skywalking集群

从github中下载skywalking的chart包仓库

1
2
3

cd /tmp
git clone https://github.com/apache/skywalking-kubernetes
cd /tmp/skywalking-kubernetes/chart

由于已存在es集群，因此不需要再通过helm去部署es。可把chart包依赖的es chart注释掉。

1	vim skywalking/Chart.yaml

执行helm命令部署skywalking集群。在第一步，我已经在kube-system名字空间下部署了es集群，因此skywalking连接的es集群是：elasticsearch-logging.kube-system:9200。

export SKYWALKING_RELEASE_NAME=skywalking
export SKYWALKING_RELEASE_NAMESPACE=skywalking
kubectl create ns $SKYWALKING_RELEASE_NAMESPACE

helm install "$SKYWALKING_RELEASE_NAME" ./skywalking \
  -n "$SKYWALKING_RELEASE_NAMESPACE" \
  --set oap.image.tag=9.1.0 \
  --set oap.storageType=elasticsearch \
  --set oap.service.type=NodePort \
  --set oap.javaOpts="-Xmx4g -Xms4g" \
  --set ui.image.tag=9.1.0 \
  --set ui.service.type=NodePort \
  --set elasticsearch.enabled=false \
  --set elasticsearch.config.host=elasticsearch-logging.elastic \
  --set elasticsearch.config.port.http=9200 \
  --set elasticsearch.config.user="" \
  --set elasticsearch.config.password=""

查看svc和pod，可见部署skywalking成功：

3.3 制作skywalking agent的init容器

mkdir skywalking-java-agent && cd skywalking-java-agent
wget  https://dlcdn.apache.org/skywalking/java-agent/8.12.0/apache-skywalking-java-agent-8.12.0.tgz
tar -xvf apache-skywalking-java-agent-8.12.0.tgz


cat > Dockerfile < EOF
FROM busybox:latest
ENV LANG=C.UTF-8
RUN set -eux && mkdir -p /opt/skywalking/agent/
ADD skywalking-agent /opt/skywalking/agent/
WORKDIR /
EOF

执行docker build命令制作镜像，并推送至仓库。
docker build -t registry.cn-hangzhou.aliyuncs.com/k8s_beijing/skywalking-agent:9.0.1 .
docker push registry.cn-hangzhou.aliyuncs.com/k8s_beijing/skywalking-agent:9.0.1

4 部署springboot微服务

1）微服务来自网上，并做了一些修改。微服务几乎没有业务逻辑，只有http调用和睡眠指令。
2）我的业务服务部署在另外一个k8s集群中，因此skywalking agent访问的是位于另一个集群中的skywalking oap服务的NodePort。
3）每个yaml文件都可以直接使用，需要根据实际情况修改环境变量SW_AGENT_COLLECTOR_BACKEND_SERVICES。在我的例子中SW_AGENT_COLLECTOR_BACKEND_SERVICES=192.9.30.230:32297。

4.1 UI服务

cat > acme-financial-ui.yaml < EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: acme-financial-ui
  name: acme-financial-ui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: acme-financial-ui
  template:
    metadata:
      labels:
        app: acme-financial-ui
    spec:
      initContainers:
      - image: registry.cn-hangzhou.aliyuncs.com/k8s_beijing/skywalking-agent:9.1.0
        name: skywalking-sidecar
        command: ["sh"]
        args: [
                "-c",
                "mkdir -p /opt/sw/agent && cp -rf /opt/skywalking/agent/* /opt/sw/agent/"
        ]
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      containers:
      - env:
        - name: JAVA_OPTS
          value: "-javaagent:/opt/sw/agent/skywalking-agent.jar"
        - name: SW_AGENT_NAME
          value: "acme-financial-ui"
        - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
          value: "192.9.30.230:32297"
        image: registry.cn-shenzhen.aliyuncs.com/gzlj/acme-financial-ui:v0.1
        imagePullPolicy: Always
        name: ui
        ports:
        - containerPort: 8081
          protocol: TCP
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      volumes:
      - name: sw-agent
        emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: acme-financial-ui
  name: acme-financial-ui
spec:
  ports:
  - name: http
    port: 8081
    protocol: TCP
    targetPort: 8081
  selector:
    app: acme-financial-ui
  sessionAffinity: None
  type: NodePort

EOF

4.2 office服务

cat > acme-financial-office.yaml < EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: acme-financial-office
  name: acme-financial-office
spec:
  replicas: 1
  selector:
    matchLabels:
      app: acme-financial-office
  template:
    metadata:
      labels:
        app: acme-financial-office
    spec:
      initContainers:
      - image: registry.cn-hangzhou.aliyuncs.com/k8s_beijing/skywalking-agent:9.1.0
        name: skywalking-sidecar
        command: ["sh"]
        args: [
                "-c",
                "mkdir -p /opt/sw/agent && cp -rf /opt/skywalking/agent/* /opt/sw/agent/"
        ]
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      containers:
      - env:
        - name: JAVA_OPTS
          value: "-javaagent:/opt/sw/agent/skywalking-agent.jar"
        - name: SW_AGENT_NAME
          value: "acme-financial-office"
        - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
          value: "192.9.30.230:32297"
        image: registry.cn-shenzhen.aliyuncs.com/gzlj/acme-financial-office:v0.1
        imagePullPolicy: Always
        name: office
        ports:
        - containerPort: 8082
          protocol: TCP
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      volumes:
      - name: sw-agent
        emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: acme-financial-office
  name: acme-financial-back-office
spec:
  ports:
  - name: http
    port: 8082
    protocol: TCP
    targetPort: 8082
  selector:
    app: acme-financial-office
  sessionAffinity: None
  type: ClusterIP 
EOF

4.3 account服务

cat > acme-financial-account.yaml < EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: acme-financial-account
  name: acme-financial-account
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: acme-financial-account
  template:
    metadata:
      labels:
        app: acme-financial-account
    spec:
      initContainers:
      - image: registry.cn-hangzhou.aliyuncs.com/k8s_beijing/skywalking-agent:9.1.0
        name: skywalking-sidecar
        command: ["sh"]
        args: [
                "-c",
                "mkdir -p /opt/sw/agent && cp -rf /opt/skywalking/agent/* /opt/sw/agent/"
        ]
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      containers:
      - env:
        - name: JAVA_OPTS
          value: "-javaagent:/opt/sw/agent/skywalking-agent.jar"
        - name: SW_AGENT_NAME
          value: "acme-financial-account"
        - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
          value: "192.9.30.230:32297"
        image: registry.cn-shenzhen.aliyuncs.com/gzlj/acme-financial-account:v0.1
        imagePullPolicy: Always
        name: account
        ports:
        - containerPort: 8083

          protocol: TCP
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      volumes:
      - name: sw-agent
        emptyDir: {}
---

apiVersion: v1
kind: Service
metadata:
  labels:
    app: acme-financial-account
  name: acme-financial-account
spec:
  ports:
  - name: http
    port: 8083
    protocol: TCP
    targetPort: 8083
  selector:
    app: acme-financial-account
  sessionAffinity: None
  type: ClusterIP

EOF

4.4 customer服务

cat > acme-financial-customer.yaml < EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: acme-financial-customer
  name: acme-financial-customer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: acme-financial-customer
  template:
    metadata:
      labels:
        app: acme-financial-customer
    spec:
      initContainers:
      - image: registry.cn-hangzhou.aliyuncs.com/k8s_beijing/skywalking-agent:9.1.0
        name: skywalking-sidecar
        command: ["sh"]
        args: [
                "-c",
                "mkdir -p /opt/sw/agent && cp -rf /opt/skywalking/agent/* /opt/sw/agent/"
        ]
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      containers:
      - env:
        - name: JAVA_OPTS
          value: "-javaagent:/opt/sw/agent/skywalking-agent.jar"
        - name: SW_AGENT_NAME
          value: "acme-financial-customer"
        - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
          value: "192.9.30.230:32297"
        image: registry.cn-shenzhen.aliyuncs.com/gzlj/acme-financial-customer:v0.1
        imagePullPolicy: Always
        name: customer
        ports:
        - containerPort: 8084
          protocol: TCP
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      volumes:
      - name: sw-agent
        emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: acme-financial-customer
  name: acme-financial-customer
  namespace: default
spec:
  ports:
  - name: http
    port: 8084
    protocol: TCP
    targetPort: 8084
  selector:
    app: acme-financial-customer
  sessionAffinity: None
  type: ClusterIP
EOF

4.5 ingress

cat > acme-ingress.yaml < EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: skywalking-ingress
  namespace: default
  annotations:
    prometheus.io/http_probe: "true"
spec:
  ingressClassName: nginx
  rules:
  - host: acme.k8s.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: acme-financial-ui
            port:
              number: 8081
EOF