上海快3官方网

追萬軟件公司-專業從事企業軟件定制系統軟件開發用戶管理系統進銷存系統等軟件開發

深入淺出時序數據庫之分布式計算

上海快3官方网WULIANWANGLINGYUJINQIRUHUORUTU,HULIANWANGHECHUANTONGGONGSIZHENGXIANGBUJUWULIANWANG。ZUOWEIWULIANWANGLINGYUSHUJUCUNCHUDESHOUXUAN,SHIXUSHUJUKUYEYUELAIYUEDUOJINRURENMENDESHIYE,ERZAOZAI2016NIAN7YUE,BAIDUYUNZAIQITIANGONGWULIANWANGPINGTAISHANGFABULEGUONEISHOUGEDUOZUHUDEFENBUSHISHIXUSHUJUKUCHANPINTSDB,CHENGWEIZHICHIQIFAZHANZHIZAO,JIAOTONG,NENGYUAN,ZHIHUICHENGSHIDENGCHANYELINGYUDEHEXINCHANPIN,TONGSHIYECHENGWEIBAIDUZHANLVEFAZHANCHANYEWULIANWANGDEBIAOZHIXINGSHIJIAN。

QIANWENTIDAOSHUJUCHAXUNTEBIESHIDASHUJULIANGDEJUHEFENXICHAXUNSHISHIXUSHUJUKUXUYAOJIEJUEDEYIGEZHUYAOWENTI,ZHIQIANDEWENZHANGJIESHAOLETONGGUOYUCHULISHUJUDEFANGFA,YONGKONGJIANHUANSHIJIANDESILU,JIANGDILEDASHUJULIANGJUHEFENXIDEYANSHI。

BENWENJIANGCONGFENBUSHIJISUANFANGXIANGSIKAO,CONGBINGFADEJIAODUJIESHAOSHIXUSHUJUKURUHEJIANGDISHUJUCHAXUNDEYANSHI。

1. 單機時序數據的聚合計算

       

WOMENXIANLAIKANKANDANJISHIRUHEZHICHIDANJUHEHANSHUDEJISUAN。DANJIJUHEJISUANFEICHANGJIANDAN,YONGHUCHAXUNSHUJUSHI,JISUANJIEDIANCHAXUNHUOQUSHIJIANFANWEINEIDESUOYOUSHIXUSHUJU,JIEDIANANZHAOSHIXUSHIYONGJUHEHANSHUDUISHUJUJINXINGJISUAN,SHENGCHENGJISUANJIEGUO。

FENXICHAXUNYEJINGCHANGHUISHIYONGQIANTAOJUHE,QIANTAOJUHEHANSHUSHIYONGBUTONGDESHIJIANCHUANGKOU,NEIBUHANSHUTONGCHANGSHIYONGXIAOSHIJIANCHUANGKOU,WAIBUSHIYONGGENGDADESHIJIANCHUANGKOU。NEIQIANTAOJUHECHAXUNZAIDANJIRUHEJISUANNI?HEDANYIJUHEHANSHULEISI,QIANTAOJUHEHANSHUDEJISUANSHIZAINEIBUJUHEHANSHUJISUANDEJIEGUOZHISHANG,GENJUSHIJIANZAICIJISUAN,HUOQUJIEGUO。RUXIATUCHAXUNYUEPINGJUNQIWENZUIDIDEYIZHOUYIJIPINGJUNQIWEN。ZONGTILAISHUO,DANJISHIXUSHUJUDEQIANTAOHEFEIQIANTAOJUHEHANSHUDESHIXIANGUOCHENGJIANDANZHIJIE,HENRONGYILIJIE。


上海快3官方网DANJIJISUANYOUSHENMETEZHENGNI?CONGDANJIDEJISUANGUOCHENG,WOMENKEYIKANDAODANJIXUYAOCHAXUNHUOQUSUOYOUYUANSHISHIXUSHUJU,YUANSHISHUJUCHAXUNDEIOCHENGBENHEJISUANCHENGBENFEICHANGGAO,ZHENGGECHAXUNDEYANSHIHUIHENGAO,DANSHIJUHEYUNSUANHOUDEJIEGUOWANGWANGSHUJULIANGHENSHAO。

2. 分布式聚合計算

上海快3官方网FENBUSHIJISUANSHIYIZHONGJISUANFANGFA,YUZHIXIANGDUIDESHIJIZHONGSHIJISUAN,SHITONGGUOSHIYONGDUOGEJISUANZIYUANZAIFENBUSHIDEHUANJINGZHONGBINGFAZHIXINGJISUANDEFANGFA。ZAISHIXUSHUJUKULINGYU,SUIZHUSHUJUDEZENGCHANG,SHIXUSHUJUHUIYUELAIYUEDUO,DANJIDECUNCHU、CHAXUNHEJUHEFENXIIOSHIJIANCHENGBENFEICHANGGAO,SUIRANSHIYONGGENGJIAGAOXIAODEYINGJIANYENENGGOUHUANJIE,DANSHIYOUCHULISHANGXIAN,JIYUCHENGBENDENGYINSUDEKAOLV,FENBUSHIJUHECHAXUNRENGRANSHISHIXUSHUJUKUZIRANERRANDEXUANZE。

上海快3官方网DANGSHIXUSHUJUKUCUNCHUDESHUJUYUELAIYUEDUOSHI,JUHECHAXUNBUKEBIMIAN,ZHEYESHIOLAPFENXICHAXUNZHONGZUICHANGJIANCAOZUOZHIYI,SHIYONGYUCHULIKEYITIGAOCHAXUNXINGNENG,DANSHIBUGOULINGHUO。FENBUSHIJUHEJISUANZESHINENGGOUSHIYONGFENBUSHIDETEXING,TONGGUODUOGEJISUANZIYUANBINGXINGJISUAN,ZAIDUIJIEGUOJINXINGHEBINGFANHUI,TONGGUOBINGFATIGAOJUHECHAXUNXINGNENG。

3. 分布式時序數據聚合計算

SHIXUSHUJUDEFENBUSHIJUHEJISUANXUYAODUOGEJIEDIANBINGXINGJISUAN,LUOJISHANGYESHIYIGEMap/ReduceDEGUOCHENG,MapGUOCHENGXUYAODUIYUANSHISHIXUSHUJUJINXINGFENPIAN,FENBIEJUHEJISUAN。ReduceGUOCHENGZESHIDUIDUOGEFENPIANJISUANJIEGUODEHEBING。WANGWANGJUHEYUNSUANDEJIEGUOHEYUANSHISHUJUYOUZHUMINGXIANSHUJULIANGDECHAJU,QICIFENBUSHIJISUANKEYIGENGDUODEKAOLVSHUJUDEBENDIHUA,YINCISHIYONGFENBUSHIJUHEJISUANXIANRANNENGGOUYOUXIAOTIGAOCHAXUNXINGNENG。

SHIXUSHUJUYAOJINXINGFENBUSHIJISUANXUYAOJIEJUELIANGGEJIBENWENTI:SHIXUSHUJUJISUANFENPIANYIJIJISUANJIEGUODEHEBING。

3.1 時序數據計算分片

SHIXUSHUJUJUHEJISUANDEFENPIANKEYIFENWEIJIGEWEIDUKAOLV:CUNCHUFENPIAN、JUHEHANSHUSHIJIANCHUANGKOUYIJICHAXUNTIAOJIAN。

SHOUXIAN,SHIXUSHUJUJUHECHAXUNBAOHANDUOZHONGTIAOJIAN,DUISHIXUSHUJUJINXINGFENZUJUHECHAXUNYESHIYIZHONGCHANGYONGCHAXUN,BUTONGDEFENZUYUANSHISHIXUSHUJUBUTONG,YINCIKEYITONGGUOCHAXUNFENZUDUISHIXUSHUJUJISUANJINXINGFENPIAN,BUTONGDEFENZUSHIYONGBUTONGJIEDIANBINGFAJISUAN。

QICI,SHIXUSHUJUJUHECHAXUNHANSHUTONGCHANGDOUBAOHANSHIJIANCHUANGKOU,XIANGTONGSHIJIANCHUANGKOUDEYUANSHISHUJUJUHEJISUANWEIYIGESHUJUDIAN,BUTONGDESHIJIANCHUANGKOUYONGYUJISUANDESHIXUYUANSHISHUJUBUTONG,YINCIYETONGYANGKEYITONGGUOSHIJIANCHUANGKOUDUISHIXUSHUJUJISUANJINXINGSHIJIANWEIDUDEFENPIAN,BUTONGDEJIEDIANJISUANBUTONGSHIJIANCHUANGKOUDESHUJU。

上海快3官方网DISAN,ANZHAOCUNCHUFENPIANJINXINGJISUAN。WOMENXIANLAIHUIYIYIXIAQIANWENSHUOMIAOSHUDESHIXUSHUJUDECUNCHU,SHIXUSHUJUYOUYUCUNCHUDESHUJULIANGHENDA,DANJIBINGBUNENGMANZUXUQIU,YINCIXUYAODUISHIXUSHUJUJINXINGFENPIANCUNCHU,FENPIAN(shard)TONGCHANGSHIYONGmetric+tagsDEFANGSHIJINXING,BUTONGDEFENPIANCUNCHUZAIBUTONGDECUNCHUJIEDIAN,FENPIANCUNCHUZHUYUANSHISHIXUSHUJU,SHIYONGCUNCHUFENPIANJINXINGFENPIANJISUAN,YESHIYIZHONGZIRANERRANDEXUANZE。RUXIATUXIANDUIshardJINXINGFENPIANJISUANCHAXUN,ZUIHOUDUIJIEGUOJINXINGHEBING。

(DIANJIFANGDATUXIANG)

上海快3官方网SHIYONGCUNCHUFENPIANLAIFENPIANJISUANYOUZHUSHENMEYOUSHINI?XIANRAN,SHUJUCHAXUNHEJISUANZAICUNCHUFENPIANDEJIEDIANSHANGJINXING,NENGGOUZUIDADEBAOZHENGSHUJUBENDIHUA,NENGGOUYOUXIAOJIANSHAOWANGLUOTONGXUNDAILAIDEYANSHI,SHIDEBENDISHUJUJISUANGENGJIAGAOXIAO。

上海快3官方网FENBUSHIJUHECHAXUNZAISHIXIANSHI,WANGWANGDUOZHONGJISUANFENPIANFANGSHITONGSHISHIYONG,JUHEJISUANJINLIANGBAOZHENGBENDIHUA、 JINLIANGDUODEBINGFAZHIXING。

3.2 時序數據計算結果的合并

上海快3官方网SHIXUSHUJUJUHEJISUANJIEGUODEHEBINGHEJISUANFENPIANDEFANGSHIYOUXIANGGUANXING,BUTONGFENPIANFANGSHIJIEGUODEHEBINGFANGSHIYEBUTONG。

SHOUXIAN,DUIYUFENZUJUHECHAXUNJIEGUODEHEBINGLAISHUO,BUTONGDEFENZUCHAXUNJIEGUOSHUYUBUTONGDEFENZU,ANZHAOFENZUJUHECHAXUNTIAOJIANHEBINGJIEGUO,JIUNENGXINGCHENGJISUANJIEGUO。

QICI,DUIYUJUHEHANSHUSHIJIANCHUANGKOUFENPIANCHAXUNDEHEBINGLAISHUO,BUTONGDESHIJIANCHUANGKOUDEJISUANJIEGUOSUIRANSHUYUTONGYIGEFENZU,DANSHIJIEGUOZAISHIJIANSHISHANGYOUXUDE,YINCIZHIXUYAODUIFENPIANJISUANJIEGUOANZHAOSHIXUPAIXUHEBING,JIUNENGHUOQUZUIZHONGJISUANJIEGUO。

DISAN,DUIYUCUNCHUFENPIANJINXINGFENPIANJISUANJIEGUODEHEBINGLAISHUO,HEBINGXIANGDUIFUZA,YINWEIZAITONGYIGESHIJIANCHUANGKOUNEI,KENENGHUIBAOHANDUOGEFENPIAN,DUOGEFENPIANSHANGTONGYISHIJIANCHUANGKOUXUYAOJUHEYUNSUANWEIYIGESHUJUDIAN。JUHEYUNSUANJIEGUODEHEBINGJIUXUYAOFENXIJUHEHANSHUDETEXINGLAIJINXING,LIRUZAIAHEBLIANGGECUNCHUFENPIANDETONGYISHIJIANCHUANGKOUNEISUMJUHEHANSHU,XIANRANJISUANJIEGUOKEYIZHIJIELEIJIASUM(A U B) = SUM(A) + SUM(B),DANSHIBINGBUSHISUOYOUDEJUHEHANSHUDOUMANZUZHEYITEXING,XUYAOGENJUJUHEHANSHUDETEXINGZUOYIYIDEFENLEI。

上海快3官方网DANGSHIYONGDUOZHONGFENPIANFANGSHIJINXINGJUHECHAXUNSHI,XIANGYINGJIEGUODEHEBINGYETONGYANGGENGWEIFUZA。

3.3 時序數據嵌套聚合運算

QIANTAOJUHECHAXUNYESHISHUJUFENXIDECHANGYONGFANGSHI,QIANTAOJUHEYUNSUANWANGWANGDUOGEJUHEHANSHUQIANTAOERCHENG,MEIGEJUHEHANSHUDEJISUANSHUXINGBINGBUWANQUANXIANGTONG。ZAIKAOLVJISUANFENPIANSHI,KEYIKAOLVJIANGWAIBUQIANTAOHANSHUHENEIBUQIANTAOHANSHUFENKAIJISUAN,XUANZEGENGJIAYOULIDEFENPIANFANGSHI。LIRUKAOLV DIFF(SUM(A, 1day)) QIANTAOJUHEHANSHU(DIFFJUHEHANSHUSHIJISUANQIANHOUSHIJIANXULIEJIEGUODECHAZHI),JIKEYISHIYONGANZHAOSHIJIANCHUANGKOUDEFANGSHIFENPIANJISUAN,YETONGYANGKEYIKAOLVJIANG DIFFDEJISUANHESUMDEJISUANCHAIFENKAILAI,XIANSHIYONGCUNCHUFENPIANDEFANGSHIJUHEJISUANSUM(A, 1day)DEJIEGUO,JIEGUOHEBINGSHIJISUANDIFFQIANTAOJUHEHANSHUDEJIEGUO,CUNCHUFENPIANDEFENBUSHIJISUANNENGGOUCHONGFENLIYONGSHUJUBENDIHUADETEXING,YINCISHIYONGHOUZHEXIANRANGENGJIAGAOXIAO。QIANTAOJUHEHANSHUDESHUJURUHEFENPIANJISUAN,XUYAOGENJUJUHEHANSHUTEXINGYIJICHANGJINGJUTIFENXI,ZHERENGRANSHIYIGEXUYAOSHENRUKAOLVDEWENTI。

3.4 計算任務的調度和優化

SHIXUSHUJUFENBUSHIJISUANCHULEJISUANFENPIANHESHUJUHEBINGWENTIYIWAI,TONGYANGXUYAOCHULIRENWUDIAODUHESQLCHAXUNYOUHUADEWENTI,XIANYOUDEHENDUOKAIYUANKUANGJIASpark、Presto、、DOUYOUXIANGYINGDEJIEJUEFANGAN,ZHELIJIUBUZUOSHENRUTAOLUNLE。

4. 時序數據聚合查詢的難題

上海快3官方网SHIXUSHUJUFENBUSHIJUHEJISUANRENGRANYOUHENDUONANTI,LIRUCOUNT(DISTINCT FIELD),ZHELEIJUHEHANSHUDETEDIANSHIZAIJISUANJIEGUOSHINEIBUXUYAOBAOCUNDALIANGDEZHONGJIANSHUJUYONGYUJISUAN,XUYAOXIAOHAODALIANGJISUANHECUNCHUZIYUAN。SUIRANHENDUODASHUJULINGYUFENBUSHICHAXUNYINQINGDENGTONGGUOSUANFADOUCHANGSHIZUOLEBUFENYOUHUA,DANSHIRENGRANWEINENGWANQUANJIEJUESUOYOUWENTI。

5. 總結

上海快3官方网ZAISHIXUSHUJUKUDASHUJULIANGJUHEFENXICHAXUNZHONG,JUHEYUNSUANZHIJIEYINGXIANGZHUCHAXUNXINGNENG,SHIYONGFENBUSHIJISUANDEFANGFA,NENGGOUYOUXIAODETIGAOCHAXUNXINGNENG,XIANGBIJIAOYUYUCHULICHAXUNGENGJIADELINGHUO。BENWENZHUYAOCONGFENPIANYIJIRUHEBINGFADEJIAODUZUOLETAOLUN,DANSHIYIXIETESHUQIANTAOJUHECHANGJINGDEYOUHUARENGJIUSHIXUYAOSHENRUSIKAOKETI。


最新文章

熱門文章