ElasticSearch数据结构
和学习数据库一样,在学习ES的时候也要注意学习数据的存储 模式结构,就能更好的理解ES的数据,如MySQL中,定义数据的最小单位是字段,也叫属性,多个字段组成一行数据,多行数据组成一张表,多张表汇聚在一个数据库下面,数据上面就是实例,即IP加端口;
ES也一样,最小单位叫field,一般叫属性,就是类似于MySQL的字段;多个field组成一个document,一般叫文档,类似于MySQL的行;多个document组成一个Type,一般叫类型,类似于MySQL上的表(版本不一样,Type略有不同,详情可参考后面的Type介绍);多个表汇总一个Index下,一般叫索引,类似于MySQL的数据库,再上面就是ES的集群地址了,一般是集群IP加ES端口号的集合;
下面简单介绍下ES的这四大结构,大致结构如图1;
以下的操作均以在kibana中操作为主,后面的Jva操作再涉及Java代码,演示用的ES版本为ES 7.7.1;
索引Index,分片和备份数
索引Index
其实就是类似于关系型数据库的的数据库的概念,数据库为了保证服务的高可用容灾,需要设置主从库等手段,ES既然是个集群也必然要保证数据的高可用和高效率,这就涉及到了ES索引的分片数和备份数;
分片数(number_of_shards):就是你买了本《西游记》,太厚了,就分为上回,中回,下回,来减轻每一本书的负担;
备份数number_of_replicas :就是你太爱看《西游记》了,于是就买了好多本一样的回来,一本丢了可以看另一本一样的,保险起见,一般不会吧两本一样的书放在一起,毕竟要丢全丢,所以备份数也一样,两份相同的备份一般不会指定在同一个节点,都是在不同的节点分开存储的,当然特殊情况下,你要是自己玩玩ES,搞个单节点的ES,那备份也就没有任何意义了;
在kibana上建立索引及制定索引的分片数和备份数如下,对应的操作关键字是PUT
;
PUT /person
{
"settings":
{
"number_of_replicas": 5,
"number_of_shards": 3,
"max_result_window":1000000
}
}
在kibana上查看索引信息如下,对应的操作关键字是GET
;
GET /person
在kibana上删除索引信息如下,对应的操作关键字是DELETE
;
DELETE /person
效果如图1,;
类型Type
类型Type
类似于关系型数据的表的概念,类型Type
这个概念有点特殊,要根据ES的版本来叙说;
- ES 5.X版本:一个索引Index下可以有多个类型Type;
- ES 6.X版本:一个索引Index下可以有1个类型Type;
- ES 7.X版本:一个索引Index下可以有0个类型Type,如果非要写的话,可以写成
_doc
;
文档Doc
文档Doc
类似于关系数据库里面的一行数据,所以文档里面包含很多列,ES把这些列叫作属性Field
,这样在建设索引Index的时候就像建表语句一样,需要建立相应的文档的mapping,这个后面会说到;
属性Field
属性Field
l类似于关系型数据库的字段的概念,一样的,每个属性有自己不同的类型,具体可以参考ES官网对于属性类型的介绍:Field datatypes;
Kibana操作ElasticSearch的文档
`以下是关于kibana操作ES文档的常用指令;
#创建索引指定属性
PUT /book
{
"settings":
{
"number_of_replicas": 5,
"number_of_shards": 3
},
"mappings":
{
"properties":
{
"name":
{
"type":"text"
},
"author":
{
"type":"keyword"
},
"count":
{
"type":"long"
},
"onSale":
{
"type":"date",
"format":"yyyy-MM-dd HH:mm:ss"
},
"descr":
{
"type":"text"
}
}
}
}
#查看索引
GET /book
#添加文档,自动生产id
POST /book/_doc
{
"name":"武侠",
"author":"咖啡",
"count":10000,
"on-sale":"2020-01-01",
"descr":"fdjfslfjsldfljsfjs"
}
#添加一个文档手动指定id,如果id=2存在,则会全部覆盖该id的属性值,该功能也可以用PUT实现
POST /book/_doc/2
{
"name":"武侠12",
"author":"咖啡1",
"count":10000,
"on-sale":"2020-01-01",
"descr":"fdjfslfjsldfljsfjs"
}
#添加一个文档手动指定id,如果id=2存在,则会全部覆盖该id的属性值,该功能同POST
PUT /book/_doc/2
{
"name":"武侠12",
"author":"咖啡1",
"count":10000,
"on-sale":"2020-01-01",
"descr":"fdjfslfjsldfljsfjs"
}
#查看文档
GET /book/_search
#修改文档,基于某一个或者几个属性,不会全部覆盖
POST /book/_update/1
{
"doc":
{
"count":22232,
"author" : "风中之神"
}
}
#删除文档
DELETE /book/_doc/1
#查询
POST /book/_search
{
"from": 0,
"size": 20,
"query":
{
"term":
{
"count":
{
"value": 10000
}
}
}
}
# 根据查询条件修改记录,将作者为咖啡,出版日期在2020-01-01的描述改成123
POST /book/_update_by_query
{
"size": 30,
"script": {
"source":"ctx._source['descr']='123'"},
"query": {
"bool": {
"must": [
{
"term": {
"author": {
"value": "咖啡"
}
}
}
,{
"term": {
"on-sale": {
"value": "2020-01-01"
}
}
}
]
}
}
}
# 添加别名
POST _aliases
{
"actions" : [{
"add" : {
"index" : "索引名" , "alias" : "别名"}}]
}
# 删除别名
POST /_aliases
{
"actions": [
{
"remove": {
"index": "索引名", "alias": "别名"}}
]
}
关于更多的查询操作,后续有篇博客细讲;
Java操作ElasticSearch
针对以上在kibana中的操作,这里咱们用Java代码也实现一遍;
Java连接ES集群
首先要做的是确保自己的Java能连接上ES集群,这里采用Maven
架构方式,所以个人的maven配置pom,xml
文件如下,针对依赖包lombok
的使用和配置,不熟悉的小伙伴可以参考下博客Java常用依赖简介——lombok;
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>es_test</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch -->
<!--ES本身的依赖-->
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>7.7.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.elasticsearch.client/elasticsearch-rest-high-level-client -->
<!--ES高级API,用来连接ES的Client等操作-->
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>7.7.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/junit/junit -->
<!--junit,Test测试使用-->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.projectlombok/lombok -->
<!--lombok ,用来自动生成对象类的构造函数,get,set属性等-->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.12</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>RELEASE</version>
<scope>compile</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind -->
<!--jackson,用来封装json-->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.11.0</version>
</dependency>
</dependencies>>
</project>
其次是最重要的步骤之一,连接ES集群,Java代码如下;
package com.ruoyin;
import jdk.nashorn.internal.parser.JSONParser;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.apache.http.HttpHost;
public class ESClient {
public static RestHighLevelClient getClient()
{
//配置集群连接的IP和端口,正式项目是要走配置文件的,这里偷懒下,就写死吧,也方便说明问题,不要骂我代码太烂就行
//创建HttpHost对象
HttpHost[] myHttpHost = new HttpHost[7];
myHttpHost[0]=new HttpHost("10.20.101.1",9200);
myHttpHost[1]=new HttpHost("10.20.101.2",9200);
myHttpHost[2]=new HttpHost("10.20.101.3",9200);
myHttpHost[3]=new HttpHost("10.20.101.4",9200);
myHttpHost[4]=new HttpHost("10.20.101.5",9200);
myHttpHost[5]=new HttpHost("10.20.101.6",9200);
myHttpHost[6]=new HttpHost("10.20.101.7",9200);
//创建RestClientBuilder对象
RestClientBuilder myRestClientBuilder=RestClient.builder(myHttpHost);
//创建RestHighLevelClient对象
RestHighLevelClient myclient=new RestHighLevelClient(myRestClientBuilder);
System.out.println("client connect ES Cluster successfully!");
return myclient;
}
}
连接ES的测试用例如下;
package test;
import com.ruoyin.ESClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.junit.Test;
public class ConES
{
@Test
public void testConES()
{
RestHighLevelClient myClient = ESClient.getClient();
}
}
Java操作ES的索引Index
这里实现的代码主要针对java实现操作ES的Index操作,涉及三个板块;
- 检查索引是否存在;
- 创建索引;
- 删除索引。
这里也是要用到前面的连接ES集群的,其实也很好理解,就是不论你做啥,首先要连接上ES才行嘛,其次,以上三种操作的具体抽象逻辑都比较类似,可以大致分为四步;
- 定义相关全局变量或对象;
- 准备request对象,不同的操作有不同的request对象,
GetIndexRequest(获取索引)
,CreateIndexRequest(创建索引)
,DeleteIndexRequest(删除索引)
; - 通过client对象执行相应的操作;
- 获取返回结果;
具体Java代码实现如下;
package test;
import com.ruoyin.ESClient;
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
import org.elasticsearch.action.support.master.AcknowledgedRequest;
import org.elasticsearch.action.support.master.AcknowledgedResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.indices.CreateIndexRequest;
import org.elasticsearch.client.indices.CreateIndexResponse;
import org.elasticsearch.client.indices.GetIndexRequest;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.json.JsonXContent;
import java.io.IOException;
import org.junit.Test;
public class CreateIndex
{
String index="person";
RestHighLevelClient myClient= ESClient.getClient();
//查询索引是否存在
@Test
public void exsitsIndex() throws IOException
{
//准备request对象
GetIndexRequest myrequest=new GetIndexRequest(index);
//通过client去操作
boolean myresult = myClient.indices().exists(myrequest,RequestOptions.DEFAULT);
//输出结果
System.out.println(myresult);
}
//创建索引
@Test
public void creatIndex() throws IOException
{
//设置索引的Setting
Settings.Builder mySettings=Settings.builder()
.put("number_of_replicas",7)
.put("number_of_shards",3);
//设置索引的mappings
XContentBuilder myMappings = JsonXContent.contentBuilder()
.startObject()
.startObject("properties")
.startObject("name")
.field("type","text")
.endObject()
.startObject("age")
.field("type","integer")
.endObject()
.startObject("birthday")
.field("type","date")
.field("format","yyyy-MM-dd")
.endObject()
.endObject()
.endObject();
//将准备好的setting和mapping封装到一个request对象内
CreateIndexRequest myrequest = new CreateIndexRequest(index)
.settings(mySettings)
.mapping(myMappings);
//通过client对象去连接ES并执行创建索引
CreateIndexResponse myCreateIndexResponse=myClient.indices().create(myrequest, RequestOptions.DEFAULT);
//输出结果
System.out.println("myCreateIndexResponse"+myCreateIndexResponse.toString());
}
//删除索引
@Test
public void deleteIndex() throws IOException {
//准备request对象
DeleteIndexRequest myDeleteIndexRequest = new DeleteIndexRequest();
myDeleteIndexRequest.indices(index);
//通过client对象执行
AcknowledgedResponse myAcknowledgedResponse = myClient.indices().delete(myDeleteIndexRequest,RequestOptions.DEFAULT);
//获取返回结果
System.out.println(myAcknowledgedResponse.isAcknowledged());
}
}
// 别名操作
public static void indexUpdateAlias(String index,String index_alias) throws IOException
{
String old_index_name="老的索引名";
log.info(index_alias+ " old index is "+old_index_name);
//删除别名映射的老的index
DeleteAliasRequest myDeleteAliasRequest = new DeleteAliasRequest(old_index_name, index_alias);
org.elasticsearch.client.core.AcknowledgedResponse myDeleteResponse=myClient.indices().deleteAlias(myDeleteAliasRequest, RequestOptions.DEFAULT);
boolean deletealisaacknowledged = myDeleteResponse.isAcknowledged();
log.info("delete index successfully? " + deletealisaacknowledged);
//新建新的index别名
IndicesAliasesRequest request = new IndicesAliasesRequest();
IndicesAliasesRequest.AliasActions aliasAction = new IndicesAliasesRequest.AliasActions(IndicesAliasesRequest.AliasActions.Type.ADD).index(index).alias(index_alias);
request.addAliasAction(aliasAction);
org.elasticsearch.action.support.master.AcknowledgedResponse indicesAliasesResponse = myClient.indices().updateAliases(request, RequestOptions.DEFAULT);
boolean createaliasacknowledged = indicesAliasesResponse.isAcknowledged();
log.info("create index successfully? "+createaliasacknowledged);
}
Java操作ES的文档(单条)
这里主要也是利用ES的高级API进行操作;
首先,因为涉及到文档的新增,删除,及修改,那么需要设置一个跟文档互相能mapping起来的对象类,即文档的每一个field对应这个对象类的一个私有变量,这里依然以Person为实验,创建对象类的代码如下;
package com.ruoyin.entity;
import com.fasterxml.jackson.annotation.JsonFormat;
import com.fasterxml.jackson.annotation.JsonIgnore;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.Date;
//定义person类,安装lobok插件和引用他的maven配置
//使用lombok后就可以根据你new对象的时候自动生成构造函数了,坑就是如果你的代码迁移了,别人没安装这个插件,会报错
@Data
@NoArgsConstructor
@AllArgsConstructor
public class Person
{
//做成json时,id不处理
@JsonIgnore
private Integer id;
private String name;
private Integer age;
//指定date的格式
@JsonFormat(pattern ="yyyy-MM-dd")
private Date birthday;
}
接下来时,以上三种操作文档的具体代码,和操作索引类似,可以大致分为四步;
- 定义相关全局变量或对象;
- 准备request对象,不同的操作有不同的request对象,
IndexRequest(创建文档请求)
,UpdateRequest(修改文档请求)
,DeleteRequest(删除文档请求)
; - 通过client对象执行相应的操作;
- 获取返回结果;
具体代码如下;
package test;
import java.io.IOException;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.ruoyin.entity.Person;
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
import org.elasticsearch.action.delete.DeleteRequest;
import org.elasticsearch.action.delete.DeleteResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.action.update.UpdateRequest;
import org.elasticsearch.action.update.UpdateResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.xcontent.XContentType;
import org.junit.Test;
import com.ruoyin.ESClient;
public class ODoc
{
//创建json的mapper对象
ObjectMapper myObjectMapper=new ObjectMapper();
//定义ES的index和client
String index="person";
//创建文档和修改文档不指定类型会报警告
String type="_doc";
RestHighLevelClient myClient= ESClient.getClient();
//添加ES的文档数据
@Test
public void createDoc() throws IOException {
//准备数据对象的json数据
Person myPerson=new Person(2,"张三",34,new Date());
String myjson =myObjectMapper.writeValueAsString(myPerson);
//准备request对象
IndexRequest myrequest= new IndexRequest(index,type,myPerson.getId().toString());
myrequest.source(myjson, XContentType.JSON);
//client操作对象添加文档
IndexResponse myIndexResponse= myClient.index(myrequest, RequestOptions.DEFAULT);
//返回结果
System.out.println(myIndexResponse.getResult().toString());
}
//修改文档
@Test
public void updateDoc() throws IOException {
//创建一个Map,指定需要修改的内容
Map<String,Object> doc = new HashMap<String, Object>();
doc.put("name","张大师");
String docID="JXePI3QBvJhVJNFwBfgY";
//创建request请求,封装数据
UpdateRequest myUpdateRequest =new UpdateRequest(index,type,docID);
myUpdateRequest.doc(doc);
//通过client对象执行
UpdateResponse myUpdateResponse=myClient.update(myUpdateRequest,RequestOptions.DEFAULT);
//输出返回结果
System.out.println(myUpdateResponse.getResult().toString());
}
//删除文档
@Test
public void deleteDoc() throws IOException {
//指定文档的id
String docID="JXePI3QBvJhVJNFwBfgY";
//创建request请求
DeleteRequest myDeleteRequest =new DeleteRequest(index,type,docID);
//通过client对象执行
DeleteResponse myDeleteResponse=myClient.delete(myDeleteRequest,RequestOptions.DEFAULT);
//输出返回结果
System.out.println(myDeleteResponse.getResult().toString());
}
}
Java操作ES的文档(批量)
Java操作ES批量新增和批量删除的核心代码非常的像,只是调用的Request类不一样了,需要调用BulkRequest
类,具体样例代码如下;
package test;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.ruoyin.ESClient;
import com.ruoyin.entity.Person;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.delete.DeleteRequest;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.RestHighLevelClient;
import org.junit.Test;
import org.elasticsearch.common.xcontent.XContentType;
import java.io.IOException;
import java.util.Date;
public class ODocs
{
//创建json的mapper对象
ObjectMapper myObjectMapper=new ObjectMapper();
//定义ES的index和client
String index="person";
//创建文档和修改文档不指定类型会报警告
String type="_doc";
RestHighLevelClient myClient= ESClient.getClient();
//批量新增数据
@Test
public void bulkCreateDOc() throws IOException {
//准备多个json数据
Person myPerson1 = new Person(1,"张三",23,new Date());
Person myPerson2 = new Person(2,"李四",27,new Date());
Person myPerson3 = new Person(3,"王五",25,new Date());
String myjson1 = myObjectMapper.writeValueAsString(myPerson1);
String myjson2 = myObjectMapper.writeValueAsString(myPerson2);
String myjson3 = myObjectMapper.writeValueAsString(myPerson3);
//创建request,将准备好
BulkRequest myBulkRequest = new BulkRequest();
myBulkRequest.add(new IndexRequest(index,type,myPerson1.getId().toString()).source(myjson1,XContentType.JSON));
myBulkRequest.add(new IndexRequest(index,type,myPerson2.getId().toString()).source(myjson2,XContentType.JSON));
myBulkRequest.add(new IndexRequest(index,type,myPerson3.getId().toString()).source(myjson3,XContentType.JSON));
//用client执行
BulkResponse myBulkResponse = myClient.bulk(myBulkRequest, RequestOptions.DEFAULT);
//输出结果
System.out.println(myBulkResponse.toString());
}
//批量删除数据
@Test
public void bulkDeleteDoc() throws IOException {
//封装request对象
BulkRequest myBulkRequest=new BulkRequest();
myBulkRequest.add(new DeleteRequest(index,type,"1"));
myBulkRequest.add(new DeleteRequest(index,type,"2"));
myBulkRequest.add(new DeleteRequest(index,type,"3"));
//client执行
BulkResponse myBulkResponse=myClient.bulk(myBulkRequest,RequestOptions.DEFAULT);
//输出响应结果
System.out.println(myBulkResponse.toString());
//因为是批处理,所以最好写完后flush下数据和refresh下数据
//插入后Refresh数据
RefreshRequest myRefreshRequest=new RefreshRequest(index);
RefreshResponse myRefreshResponse = myClient.indices().refresh(myRefreshRequest,RequestOptions.DEFAULT);
System.out.println(myRefreshResponse.getStatus());
//插入后Flush数据
FlushRequest myFlushRequest=new FlushRequest(index);
FlushResponse myFlushResponse= myClient.indices().flush(myFlushRequest,RequestOptions.DEFAULT);
System.out.println(myFlushResponse.getStatus());
}
}
以上Java代码的操作,重点都是在实现操作ES的核心代码,关于其他技术跟ES的结合,以及ES的查询,后面博客再做叙述;
本文参考链接:https://blog.csdn.net/LXWalaz1s1s/article/details/108141731