缓存数据生产服务的工作流程分析以及工程环境搭建

发布 : 2017-07-02 分类 : 大数据 浏览 :

需求

1
2
3
专门做一个缓存数据生产服务，去不断的监听各种缓存数据对应的原始服务
有没有数据变更的消息
如果有，就去重新生产缓存数据，更新缓存数据

缓存清洗策略

1
2
(1).Redis:LRU策略
(2).nginx本地缓存，一般采取主动过期，每隔10分钟，就自动过期

商品详情页缓存数据生产服务的工作流程分析

1
2
3
4
5
6
7
8
9
三级缓存，多级缓存是由：

服务本地堆缓存 + redis分布式缓存 + nginx本地缓存组成的

综合利用多级的缓存，才能支撑住高并发场景下各种特殊情况

服务本地堆缓存的作用是预防redis层的彻底崩溃，作为缓存的最后一道防线，避免数据库直接裸奔

服务本地堆缓存是用java最流行的缓存的框架ehcache做缓存

工作流程

1
2
3
4
5
(1).监听多个kafka topic，每个kafka topic对应一个服务
(2).如果一个服务发生了数据变更，那么就发送一个消息到kafka topic中
(3).缓存数据生产服务监听到了消息以后，就发送请求到对应的服务中调用接口以及拉取数据，此时是从mysql中查询的
(4).缓存数据生产服务拉取到了数据之后，会将数据在本地缓存中写入一份，就是ehcache中
(5).同时会将数据在redis中写入一份

Spring boot+Mybatis+ehcache+Redis框架整合搭建

pom.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.matrix.eshop</groupId>
	<artifactId>eshop-cache</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>jar</packaging>

	<name>eshop-cache</name>
	<url>http://maven.apache.org</url>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.2.5.RELEASE</version>
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-thymeleaf</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-jdbc</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>
		<dependency>
			<groupId>org.mybatis</groupId>
			<artifactId>mybatis-spring</artifactId>
			<version>1.2.2</version>
		</dependency>
		<dependency>
			<groupId>org.mybatis</groupId>
			<artifactId>mybatis</artifactId>
			<version>3.2.8</version>
		</dependency>
		<dependency>
			<groupId>org.apache.tomcat</groupId>
			<artifactId>tomcat-jdbc</artifactId>
		</dependency>
		<dependency>
			<groupId>mysql</groupId>
			<artifactId>mysql-connector-java</artifactId>
		</dependency>
		<dependency>
			<groupId>com.alibaba</groupId>
			<artifactId>fastjson</artifactId>
			<version>1.1.43</version>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>

	<repositories>
		<repository>
			<id>spring-milestone</id>
			<url>https://repo.spring.io/libs-release</url>
		</repository>
	</repositories>

	<pluginRepositories>
		<pluginRepository>
			<id>spring-milestone</id>
			<url>https://repo.spring.io/libs-release</url>
		</pluginRepository>
	</pluginRepositories>
</project>

Application.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
package com.matrix.eshop.cache;

import org.apache.ibatis.session.SqlSessionFactory;
import org.apache.tomcat.jdbc.pool.DataSource;
import org.mybatis.spring.SqlSessionFactoryBean;
import org.mybatis.spring.annotation.MapperScan;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.transaction.PlatformTransactionManager;

/**
 * 库存服务的Application入口类
 *
 * @author matrix
 *
 */
@EnableAutoConfiguration // 注解
@SpringBootApplication // 启动类
@ComponentScan // 开启组件扫描
@MapperScan("com.matrix.eshop.cache.mapper")
public class Application {

	/**
	 * 构建数据源
	 *
	 * @param args
	 */
	@Bean
	@ConfigurationProperties("spring.datasource")
	public DataSource dataSource() {
		return new org.apache.tomcat.jdbc.pool.DataSource();
	}

	/**
	 * 构建Mybatis入口类
	 *
	 * @return
	 * @throws Exception
	 */
	@Bean
	public SqlSessionFactory sqlSessionFactory() throws Exception {
		SqlSessionFactoryBean sqlSessionFactoryBean = new SqlSessionFactoryBean();
		sqlSessionFactoryBean.setDataSource(dataSource());
		PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
		sqlSessionFactoryBean.setMapperLocations(resolver.getResources("classpath:/mybatis/*.xml"));
		return sqlSessionFactoryBean.getObject();
	}

	/**
	 * 构建事务管理器
	 *
	 * @return
	 */
	@Bean
	public PlatformTransactionManager transactionManager() {
		return new DataSourceTransactionManager(dataSource());
	}

	/**
	 * Spring Boot Application启动入口main方法
	 *
	 * @param args
	 */
	public static void main(String[] args) {
		SpringApplication.run(Application.class, args);
	}
}

Application.properties

1
2
3
4
spring.datasource.url=jdbc:mysql://192.168.31.234:3306/eshop
spring.datasource.username=eshop
spring.datasource.password=123456
spring.datasource.driver-class-name=com.mysql.jdbc.Driver

整合Jedis Cluster

pom.xml

1
2
3
4
<dependency>
    <groupId>redis.clients</groupId>
    <artifactId>jedis</artifactId>
</dependency>

Application.java

1
2
3
4
5
6
7
8
9
@Bean
public JedisCluster JedisClusterFactory() {
	Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
	jedisClusterNodes.add(new HostAndPort("192.168.31.231", 7003));
	jedisClusterNodes.add(new HostAndPort("192.168.31.232", 7004));
	jedisClusterNodes.add(new HostAndPort("192.168.31.233", 7006));
	JedisCluster jedisCluster = new JedisCluster(jedisClusterNodes);
	return jedisCluster;
}

整合ehcache

pom.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-context-support</artifactId>
</dependency>
<dependency>
  <groupId>net.sf.ehcache</groupId>
  <artifactId>ehcache</artifactId>
  <version>2.8.3</version>
</dependency>

缓存配置管理类

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
package com.matrix.eshop.cache.configuration;

import org.springframework.cache.annotation.EnableCaching;
import org.springframework.cache.ehcache.EhCacheCacheManager;
import org.springframework.cache.ehcache.EhCacheManagerFactoryBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;

/**
 * 缓存配置管理类
 *
 * @author matrix
 *
 */
@Configuration
@EnableCaching
public class CacheConfiguration {

	@Bean
	public EhCacheManagerFactoryBean ehCacheManagerFactoryBean() {
		EhCacheManagerFactoryBean ehCacheManagerFactoryBean = new EhCacheManagerFactoryBean();
		ehCacheManagerFactoryBean.setConfigLocation(new ClassPathResource("ehcache.xml"));
		ehCacheManagerFactoryBean.setShared(true);
		return ehCacheManagerFactoryBean;
	}

	@Bean
	public EhCacheCacheManager ehCacheCacheManager(EhCacheManagerFactoryBean ehCacheManagerFactoryBean){
		return new EhCacheCacheManager(ehCacheManagerFactoryBean.getObject());
	}
}

ehcache.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<?xml version="1.0" encoding="UTF-8"?>
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd"
	updateCheck="false">

	<diskStore path="java.io.tmpdir/Tmp_EhCache" />

	<defaultCache eternal="false" maxElementsInMemory="1000"
		overflowToDisk="false" diskPersistent="false" timeToIdleSeconds="0"
		timeToLiveSeconds="0" memoryStoreEvictionPolicy="LRU" />

	<cache name="local" eternal="false" maxElementsInMemory="1000"
		overflowToDisk="false" diskPersistent="false" timeToIdleSeconds="0"
		timeToLiveSeconds="0" memoryStoreEvictionPolicy="LRU" />

</ehcache>

Model

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
package com.matrix.eshop.cache.model;

/**
 * 商品信息
 *
 * @author matrix
 *
 */
public class ProductInfo {

	private Long id;
	private String name;
	private Double price;

	public ProductInfo() {

	}

	public ProductInfo(Long id, String name, Double price) {
		this.id = id;
		this.name = name;
		this.price = price;
	}

	public Long getId() {
		return id;
	}

	public void setId(Long id) {
		this.id = id;
	}

	public String getName() {
		return name;
	}

	public void setName(String name) {
		this.name = name;
	}

	public Double getPrice() {
		return price;
	}

	public void setPrice(Double price) {
		this.price = price;
	}

}

CacheService

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
package com.matrix.eshop.cache.service;

import com.matrix.eshop.cache.model.ProductInfo;

/**
 * 缓存service接口
 * @author matrix
 *
 */
public interface CacheService {

	/**
	 * 将商品信息保存到本地缓存中
	 * @param productInfo
	 * @return
	 */
	public ProductInfo saveLocalCache(ProductInfo productInfo);

	/**
	 * 从本地缓存中获取商品信息
	 * @param id
	 * @return
	 */
	public ProductInfo getLocalCache(Long id);
}

CacheServiceImpl

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
package com.matrix.eshop.cache.service.impl;

import org.springframework.cache.annotation.CachePut;
import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;

import com.matrix.eshop.cache.model.ProductInfo;
import com.matrix.eshop.cache.service.CacheService;

/**
 * 缓存Service实现类
 *
 * @author matrix
 *
 */
@Service("cacheService")
public class CacheServiceImpl implements CacheService {

	public static final String CACHE_NAME = "local";

	/**
	 * 将商品信息保存到本地缓存中
	 *
	 * @param productInfo
	 * @return
	 */
	@CachePut(value = CACHE_NAME, key = "'key_'+#productInfo.getId()")
	public ProductInfo saveLocalCache(ProductInfo productInfo) {
		return productInfo;
	}

	/**
	 * 从本地缓存中获取商品信息
	 *
	 * @param id
	 * @return
	 */
	@Cacheable(value = CACHE_NAME, key = "'key_'+#id")
	public ProductInfo getLocalCache(Long id) {
		return null;
	}

}

测试ehcache的整合

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
package com.matrix.eshop.cache.controller;

import javax.annotation.Resource;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;

import com.matrix.eshop.cache.model.ProductInfo;
import com.matrix.eshop.cache.service.CacheService;

/**
 * 缓存Controller
 *
 * @author matrix
 *
 */
@Controller
public class CacheController {

	@Resource
	private CacheService cacheService;

	@RequestMapping("/testPutCache")
	@ResponseBody
	public String testPutCache(ProductInfo productInfo) {
		cacheService.saveLocalCache(productInfo);
		return "success";
	}

	@RequestMapping("/testGetCache")
	@ResponseBody
	public ProductInfo testGetCache(Long id) {
		return cacheService.getLocalCache(id);
	}
}

浏览器地址栏中输入

1
http://localhost:8080/testPutCache?id=1&name=test_product&price=5555

1
http://localhost:8080/testGetCache?id=1

本文作者 : Matrix
原文链接 : https://matrixsparse.github.io/2017/07/02/缓存数据生产服务的工作流程分析以及工程环境搭建/
版权声明 : 本博客所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明出处！

知识 & 情怀 | 二者兼得

微信扫一扫, 向我投食

微信扫一扫, 向我投食

支付宝扫一扫, 向我投食

支付宝扫一扫, 向我投食

标签: #大数据 #Redis
留下足迹