Summary¶

This note introduces the NLFFF dataset and the basic use of the pynlfff toolkit.

Description¶

The toolkit is divided into two parts: dataset construction and product usage.

Dataset construction¶

  • There is a cpu version of dataset builder, you can find the dataset builder related program

at pynlfff github

3D magnetic field data products using¶

  • Product usage is divided into two parts: basic usage and toolkit-based usage.

  • We have distilled the basic usage in the toolkit to help researchers who are not comfortable downloading the full toolkit and who wish to borrow ideas to implement it through other languages.

  • We have provided some slightly more advanced features based on the use of the toolkit, such as visualization, etc. More features will be added later.

notebook can be click here to download, or click here to open it in colab, and the html version can be viewed here

Data sets and labels¶

Download dataset summary information¶

In [ ]:
# Download dataset summary information
!wget https://figshare.com/ndownloader/files/39406012 -O dataset_info.zip

--2023-02-27 17:40:58--  https://figshare.com/ndownloader/files/39406012
正在解析主机 figshare.com (figshare.com)... 54.217.34.18, 34.252.222.205, 2a05:d018:1f4:d000:647c:7301:fad1:b2b9, ...
正在连接 figshare.com (figshare.com)|54.217.34.18|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 302 Found
位置：https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/39406012/nlfffdatasetinfo.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIYCQYOYV5JSSROOA/20230227/eu-west-1/s3/aws4_request&X-Amz-Date=20230227T094059Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=a3b7af9f8a0727b36a17a2762fd496a814e6f918790fadbe56457592f125c600 [跟随至新的 URL]
--2023-02-27 17:40:59--  https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/39406012/nlfffdatasetinfo.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIYCQYOYV5JSSROOA/20230227/eu-west-1/s3/aws4_request&X-Amz-Date=20230227T094059Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=a3b7af9f8a0727b36a17a2762fd496a814e6f918790fadbe56457592f125c600
正在解析主机 s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 52.218.88.91, 52.218.90.131, 52.218.49.36, ...
正在连接 s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|52.218.88.91|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：166327650 (159M) [application/zip]
正在保存至: “dataset_info.zip”

dataset_info.zip    100%[===================>] 158.62M  2.77MB/s  用时 55s       

2023-02-27 17:41:56 (2.86 MB/s) - 已保存 “dataset_info.zip” [166327650/166327650])
In [ ]:
# Unpacking Dataset Summary Information
!unzip dataset_info.zip

Archive:  dataset_info.zip
   creating: nlfff-dataset-info/
  inflating: nlfff-dataset-info/README.txt  
   creating: nlfff-dataset-info/archive-202203/
  inflating: nlfff-dataset-info/archive-202203/archive-202203-info.csv  
  inflating: nlfff-dataset-info/changelog.txt  
  inflating: nlfff-dataset-info/config.json  
   creating: nlfff-dataset-info/demo/
   creating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/Bout.bin  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/Energy.log  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/NLFFFquality1.log  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/NLFFFquality2.log  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/NLFFFquality3.log  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/allboundaries1.dat  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/allboundaries2.dat  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/allboundaries3.dat  
 extracting: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/boundary.ini  
 extracting: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/grid1.ini  
 extracting: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/grid2.ini  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/grid2n2.ini  
 extracting: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/grid3.ini  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/mask1.dat  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/mask2.dat  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/mask3.dat  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/prot1.log  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/prot2.log  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/prot3.log  
   creating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/raw/
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/raw/hmi.sharp_cea_720s.7327.20181220_093600_TAI.Bp.fits  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/raw/hmi.sharp_cea_720s.7327.20181220_093600_TAI.Br.fits  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/raw/hmi.sharp_cea_720s.7327.20181220_093600_TAI.Bt.fits  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/run.log  
 extracting: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/step1.log  
 extracting: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/step2.log  
 extracting: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/step3.log  
  inflating: nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/zboundaries2n2.dat  
  inflating: nlfff-dataset-info/python_guide_en.ipynb  
  inflating: nlfff-dataset-info/python_guide_zh.ipynb  
   creating: nlfff-dataset-info/raw-202203/
  inflating: nlfff-dataset-info/raw-202203/raw-202203-info.csv

Download label Information¶

In [ ]:
!wget https://figshare.com/ndownloader/files/39406015 -O dataset_label.zip

--2023-02-27 17:41:59--  https://figshare.com/ndownloader/files/39406015
正在解析主机 figshare.com (figshare.com)... 34.252.222.205, 54.217.34.18, 2a05:d018:1f4:d000:647c:7301:fad1:b2b9, ...
正在连接 figshare.com (figshare.com)|34.252.222.205|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 302 Found
位置：https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/39406015/nlfffdatasetflarelabel.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIYCQYOYV5JSSROOA/20230227/eu-west-1/s3/aws4_request&X-Amz-Date=20230227T094200Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=2c5d5e692918327678bb12355682195f18fbaaa46291490820674841e3f8f242 [跟随至新的 URL]
--2023-02-27 17:42:00--  https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/39406015/nlfffdatasetflarelabel.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIYCQYOYV5JSSROOA/20230227/eu-west-1/s3/aws4_request&X-Amz-Date=20230227T094200Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=2c5d5e692918327678bb12355682195f18fbaaa46291490820674841e3f8f242
正在解析主机 s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 52.218.53.234, 52.92.34.48, 52.92.0.56, ...
正在连接 s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|52.218.53.234|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：3249636 (3.1M) [application/zip]
正在保存至: “dataset_label.zip”

dataset_label.zip   100%[===================>]   3.10M  1.19MB/s  用时 2.6s      

2023-02-27 17:42:04 (1.19 MB/s) - 已保存 “dataset_label.zip” [3249636/3249636])
In [ ]:
!unzip dataset_label.zip

Archive:  dataset_label.zip
   creating: nlfff-dataset-flare-label/
  inflating: nlfff-dataset-flare-label/README.txt  
  inflating: nlfff-dataset-flare-label/changelog.txt  
  inflating: nlfff-dataset-flare-label/config.json  
   creating: nlfff-dataset-flare-label/label-202203/
   creating: nlfff-dataset-flare-label/label-202203/prepare/
  inflating: nlfff-dataset-flare-label/label-202203/prepare/knoaa_vflaretimelist.pickle  
  inflating: nlfff-dataset-flare-label/label-202203/prepare/ksharp_vnoaa.pickle  
   creating: nlfff-dataset-flare-label/label-202203/process/
  inflating: nlfff-dataset-flare-label/label-202203/process/label.csv  
  inflating: nlfff-dataset-flare-label/label-202203/process/label_col_description.jpg  
   creating: nlfff-dataset-flare-label/label-202203/process/label_failed/
  inflating: nlfff-dataset-flare-label/label-202203/process/label_failed/no_flare_info_from_noaa_ars.txt  
  inflating: nlfff-dataset-flare-label/label-202203/process/label_failed/no_noaalist_from_sharpnum.txt  
   creating: nlfff-dataset-flare-label/label-202203/raw_file/
  inflating: nlfff-dataset-flare-label/label-202203/raw_file/all_harps_with_noaa_ars.txt  
  inflating: nlfff-dataset-flare-label/label-202203/raw_file/flare_info_processed.csv  
  inflating: nlfff-dataset-flare-label/label-202203/raw_file/flare_info_processed_col_description.jpg  
  inflating: nlfff-dataset-flare-label/python_guide_en.ipynb  
  inflating: nlfff-dataset-flare-label/python_guide_zh.ipynb

Download database archive information¶

In [ ]:
!wget https://figshare.com/ndownloader/files/39406063 -O dataset_archive.zip

--2023-02-27 17:42:05--  https://figshare.com/ndownloader/files/39406063
正在解析主机 figshare.com (figshare.com)... 34.252.222.205, 54.217.34.18, 2a05:d018:1f4:d000:647c:7301:fad1:b2b9, ...
正在连接 figshare.com (figshare.com)|34.252.222.205|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 302 Found
位置：https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/39406063/nlfffdatasetandflarelabeldbarchive.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIYCQYOYV5JSSROOA/20230227/eu-west-1/s3/aws4_request&X-Amz-Date=20230227T094206Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=d5f26997359e1d43f48ddb1487f125deba286eb72fba3b7d6d1c70e5c4ad0740 [跟随至新的 URL]
--2023-02-27 17:42:06--  https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/39406063/nlfffdatasetandflarelabeldbarchive.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIYCQYOYV5JSSROOA/20230227/eu-west-1/s3/aws4_request&X-Amz-Date=20230227T094206Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=d5f26997359e1d43f48ddb1487f125deba286eb72fba3b7d6d1c70e5c4ad0740
正在解析主机 s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 52.218.53.234, 52.218.57.243, 52.92.34.216, ...
正在连接 s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|52.218.53.234|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：85650805 (82M) [application/zip]
正在保存至: “dataset_archive.zip”

dataset_archive.zip 100%[===================>]  81.68M  2.13MB/s  用时 40s       

2023-02-27 17:42:47 (2.06 MB/s) - 已保存 “dataset_archive.zip” [85650805/85650805])
In [ ]:
!unzip dataset_archive.zip

Archive:  dataset_archive.zip
   creating: nlfff-dataset-and-flare-label-db-archive/
  inflating: nlfff-dataset-and-flare-label-db-archive/README.txt  
  inflating: nlfff-dataset-and-flare-label-db-archive/changelog.txt  
  inflating: nlfff-dataset-and-flare-label-db-archive/config.json  
  inflating: nlfff-dataset-and-flare-label-db-archive/python_guide_en.ipynb  
  inflating: nlfff-dataset-and-flare-label-db-archive/python_guide_zh.ipynb  
   creating: nlfff-dataset-and-flare-label-db-archive/sqlite-202203/
  inflating: nlfff-dataset-and-flare-label-db-archive/sqlite-202203/nlfff-202203
In [ ]:
nlfff_path_dataset_summary="nlfff-dataset-info"
nlfff_path_dataset_label="nlfff-dataset-flare-label"
nlfff_path_dataset_archive="nlfff-dataset-and-flare-label-db-archive"

Using dataset summary information¶

Read configuration¶

In [ ]:
import os
import json

# Read configuration file
nlfff_path_dataset_summary="nlfff-dataset-info"
config_path=os.path.join(nlfff_path_dataset_summary,"config.json")
with open(config_path,"r") as f:
    config_ds=json.loads(f.read())
In [ ]:
config_ds
Out[ ]:
{'config_version': 1.3,
 'raw_summary_info_path': 'raw-202203/raw-202203-info.csv',
 'product_summary_info_path': 'archive-202203/archive-202203-info.csv',
 'demo_sample': [{'harpnum_trec': '7327.20181220_093600_TAI',
   'path': 'demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI'}]}

Read raw header file information¶

In [ ]:
import pandas
import os

raw_info_path=os.path.join(
  nlfff_path_dataset_summary,
  config_ds["raw_summary_info_path"]
)
raw_info=pandas.read_csv(raw_info_path)

/home/zander/anaconda3/lib/python3.9/site-packages/IPython/core/interactiveshell.py:3444: DtypeWarning: Columns (145,254) have mixed types.Specify dtype option on import or set low_memory=False.
  exec(code_obj, self.user_global_ns, self.user_ns)
In [ ]:
raw_info
Out[ ]:
harpnum_trec HARP_NUM T_REC_DATETIME T_REC_STR FILE_NAME query DATE DATE_S DATE_B DATE__OBS ... continuum Bp Bt Br Bp_err Bt_err Br_err conf_disambig BHARP MHARP
0 1644.20120516_031200_TAI 1644 2012-05-16 03:12:00.000 20120516_031200_TAI hmi.sharp_cea_720s.1644.20120516_031200_TAI hmi.sharp_cea_720s[1644][2012.05.16_03:12:00_TAI] 2018-05-09T03:05:47Z 2012-05-20T19:22:46Z 2015-09-30T20:47:04Z 2012-05-16T03:10:24.10Z ... /SUM90/D1042885471/S00000/continuum.fits /SUM90/D1042885471/S00000/Bp.fits /SUM90/D1042885471/S00000/Bt.fits /SUM90/D1042885471/S00000/Br.fits /SUM90/D1042885471/S00000/Bp_err.fits /SUM90/D1042885471/S00000/Bt_err.fits /SUM90/D1042885471/S00000/Br_err.fits /SUM90/D1042885471/S00000/conf_disambig.fits InvalidLink hmi.Mharp_720s[1644][2012.05.16_03:12:00_TAI]
1 4315.20140708_013600_TAI 4315 2014-07-08 01:36:00.000 20140708_013600_TAI hmi.sharp_cea_720s.4315.20140708_013600_TAI hmi.sharp_cea_720s[4315][2014.07.08_01:36:00_TAI] 2014-08-13T03:40:53Z 2014-07-12T23:18:52Z 2014-07-16T08:03:00Z 2014-07-08T01:34:25.80Z ... /SUM22/D597857266/S00000/continuum.fits /SUM22/D597857266/S00000/Bp.fits /SUM22/D597857266/S00000/Bt.fits /SUM22/D597857266/S00000/Br.fits /SUM22/D597857266/S00000/Bp_err.fits /SUM22/D597857266/S00000/Bt_err.fits /SUM22/D597857266/S00000/Br_err.fits /SUM22/D597857266/S00000/conf_disambig.fits InvalidLink hmi.Mharp_720s[4315][2014.07.08_01:36:00_TAI]
2 4351.20140714_142400_TAI 4351 2014-07-14 14:24:00.000 20140714_142400_TAI hmi.sharp_cea_720s.4351.20140714_142400_TAI hmi.sharp_cea_720s[4351][2014.07.14_14:24:00_TAI] 2014-08-19T03:03:14Z 2014-07-18T23:26:26Z 2014-07-22T00:47:31Z 2014-07-14T14:22:25.70Z ... /SUM1/D599938236/S00000/continuum.fits /SUM1/D599938236/S00000/Bp.fits /SUM1/D599938236/S00000/Bt.fits /SUM1/D599938236/S00000/Br.fits /SUM1/D599938236/S00000/Bp_err.fits /SUM1/D599938236/S00000/Bt_err.fits /SUM1/D599938236/S00000/Br_err.fits /SUM1/D599938236/S00000/conf_disambig.fits InvalidLink hmi.Mharp_720s[4351][2014.07.14_14:24:00_TAI]
3 536.20110426_031200_TAI 536 2011-04-26 03:12:00.000 20110426_031200_TAI hmi.sharp_cea_720s.536.20110426_031200_TAI hmi.sharp_cea_720s[536][2011.04.26_03:12:00_TAI] 2018-04-29T14:50:56Z 2012-09-05T09:08:03Z 2015-05-12T15:15:34Z 2011-04-26T03:10:21.60Z ... /SUM90/D1038869818/S00000/continuum.fits /SUM90/D1038869818/S00000/Bp.fits /SUM90/D1038869818/S00000/Bt.fits /SUM90/D1038869818/S00000/Br.fits /SUM90/D1038869818/S00000/Bp_err.fits /SUM90/D1038869818/S00000/Bt_err.fits /SUM90/D1038869818/S00000/Br_err.fits /SUM90/D1038869818/S00000/conf_disambig.fits InvalidLink hmi.Mharp_720s[536][2011.04.26_03:12:00_TAI]
4 5415.20150409_080000_TAI 5415 2015-04-09 08:00:00.000 20150409_080000_TAI hmi.sharp_cea_720s.5415.20150409_080000_TAI hmi.sharp_cea_720s[5415][2015.04.09_08:00:00_TAI] 2015-05-15T03:15:44Z 2015-04-13T14:07:11Z 2015-04-14T18:23:45Z 2015-04-09T07:58:18.30Z ... /SUM35/D698134494/S00000/continuum.fits /SUM35/D698134494/S00000/Bp.fits /SUM35/D698134494/S00000/Bt.fits /SUM35/D698134494/S00000/Br.fits /SUM35/D698134494/S00000/Bp_err.fits /SUM35/D698134494/S00000/Bt_err.fits /SUM35/D698134494/S00000/Br_err.fits /SUM35/D698134494/S00000/conf_disambig.fits InvalidLink hmi.Mharp_720s[5415][2015.04.09_08:00:00_TAI]
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
73742 6258.20160115_222400_TAI 6258 2016-01-15 22:24:00.000 20160115_222400_TAI hmi.sharp_cea_720s.6258.20160115_222400_TAI hmi.sharp_cea_720s[6258][2016.01.15_22:24:00_TAI] 2016-02-20T04:08:10Z 2016-01-19T17:42:19Z 2016-01-20T19:17:24Z 2016-01-15T22:22:08.30Z ... /SUM80/D789029067/S00000/continuum.fits /SUM80/D789029067/S00000/Bp.fits /SUM80/D789029067/S00000/Bt.fits /SUM80/D789029067/S00000/Br.fits /SUM80/D789029067/S00000/Bp_err.fits /SUM80/D789029067/S00000/Bt_err.fits /SUM80/D789029067/S00000/Br_err.fits /SUM80/D789029067/S00000/conf_disambig.fits InvalidLink hmi.Mharp_720s[6258][2016.01.15_22:24:00_TAI]
73743 6599.20160607_080000_TAI 6599 2016-06-07 08:00:00.000 20160607_080000_TAI hmi.sharp_cea_720s.6599.20160607_080000_TAI hmi.sharp_cea_720s[6599][2016.06.07_08:00:00_TAI] 2017-12-05T18:45:13Z 2017-11-04T07:17:18Z 2017-11-26T22:59:24Z 2016-06-07T07:58:46.60Z ... /SUM95/D996809863/S00000/continuum.fits /SUM95/D996809863/S00000/Bp.fits /SUM95/D996809863/S00000/Bt.fits /SUM95/D996809863/S00000/Br.fits /SUM95/D996809863/S00000/Bp_err.fits /SUM95/D996809863/S00000/Bt_err.fits /SUM95/D996809863/S00000/Br_err.fits /SUM95/D996809863/S00000/conf_disambig.fits InvalidLink hmi.Mharp_720s[6599][2016.06.07_08:00:00_TAI]
73744 7255.20180512_080000_TAI 7255 2018-05-12 08:00:00.000 20180512_080000_TAI hmi.sharp_cea_720s.7255.20180512_080000_TAI hmi.sharp_cea_720s[7255][2018.05.12_08:00:00_TAI] 2018-06-17T03:05:37Z 2018-05-16T20:22:23Z 2018-05-18T07:52:28Z 2018-05-12T07:58:43.20Z ... /SUM80/D1055843035/S00000/continuum.fits /SUM80/D1055843035/S00000/Bp.fits /SUM80/D1055843035/S00000/Bt.fits /SUM80/D1055843035/S00000/Br.fits /SUM80/D1055843035/S00000/Bp_err.fits /SUM80/D1055843035/S00000/Bt_err.fits /SUM80/D1055843035/S00000/Br_err.fits /SUM80/D1055843035/S00000/conf_disambig.fits InvalidLink hmi.Mharp_720s[7255][2018.05.12_08:00:00_TAI]
73745 892.20110926_080000_TAI 892 2011-09-26 08:00:00.000 20110926_080000_TAI hmi.sharp_cea_720s.892.20110926_080000_TAI hmi.sharp_cea_720s[892][2011.09.26_08:00:00_TAI] 2018-04-28T01:47:53Z 2011-09-30T18:34:03Z 2015-07-02T20:56:15Z 2011-09-26T07:58:20.00Z ... /SUM87/D1038232342/S00000/continuum.fits /SUM87/D1038232342/S00000/Bp.fits /SUM87/D1038232342/S00000/Bt.fits /SUM87/D1038232342/S00000/Br.fits /SUM87/D1038232342/S00000/Bp_err.fits /SUM87/D1038232342/S00000/Bt_err.fits /SUM87/D1038232342/S00000/Br_err.fits /SUM87/D1038232342/S00000/conf_disambig.fits InvalidLink hmi.Mharp_720s[892][2011.09.26_08:00:00_TAI]
73746 1422.20120301_160000_TAI 1422 2012-03-01 16:00:00.000 20120301_160000_TAI hmi.sharp_cea_720s.1422.20120301_160000_TAI hmi.sharp_cea_720s[1422][2012.03.01_16:00:00_TAI] 2018-04-29T18:00:12Z 2012-09-05T09:13:59Z 2015-09-01T05:38:35Z 2012-03-01T15:58:13.90Z ... /SUM94/D1038926095/S00000/continuum.fits /SUM94/D1038926095/S00000/Bp.fits /SUM94/D1038926095/S00000/Bt.fits /SUM94/D1038926095/S00000/Br.fits /SUM94/D1038926095/S00000/Bp_err.fits /SUM94/D1038926095/S00000/Bt_err.fits /SUM94/D1038926095/S00000/Br_err.fits /SUM94/D1038926095/S00000/conf_disambig.fits InvalidLink hmi.Mharp_720s[1422][2012.03.01_16:00:00_TAI]

73747 rows × 276 columns

Read product file information¶

Record some basic information about the 3D magnetic field database products

In [ ]:
product_info_path=os.path.join(
  nlfff_path_dataset_summary,
  config_ds["product_summary_info_path"]
)
product_info=pandas.read_csv(product_info_path)
In [ ]:
product_info
Out[ ]:
harpnum_trec sharp_num t_rec bout_path bout_md5 bout_size grid_x grid_y grid_z grid_xyz bout_maxlevel bout_quality batch notes bout_quality_value identifiers
0 6558.20160520_044800_TAI 6558 2016-05-20 04:48:00.000 grid3.done/num_6500_6599/hmi.sharp_cea_720s.65... ea4c960822a1284fe43637b1717bc30b 3580360704 432 764 452 149181696 3 True 20220301 new 15.568000 6558201605200448003
1 6558.20160520_044800_TAI 6558 2016-05-20 04:48:00.000 grid2.done/hmi.sharp_cea_720s.6558.20160520_04... bf41326b2b90597a539a796c7eb72e10 447545088 216 382 226 18647712 2 True 20220301 old 16.364700 6558201605200448002
2 2887.20130629_111200_TAI 2887 2013-06-29 11:12:00.000 grid2.done/hmi.sharp_cea_720s.2887.20130629_11... c3661aaef993bed04a10fb74483e7613 456693120 214 390 228 19028880 2 True 20220301 old 17.953400 2887201306291112002
3 2887.20130629_111200_TAI 2887 2013-06-29 11:12:00.000 grid3.done/num_2800_2899/hmi.sharp_cea_720s.28... 38f40bf4cd8e7fa66d2563570ccbbd3f 3653544960 428 780 456 152231040 3 True 20220301 new 18.468100 2887201306291112003
4 5183.20150215_080000_TAI 5183 2015-02-15 08:00:00.000 grid2.done/hmi.sharp_cea_720s.5183.20150215_08... 7de9b081558f9069f4a4732f8bdcb42f 198342144 168 286 172 8264256 2 True 20220301 old 17.937401 5183201502150800002
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
73742 4616.20141003_142400_TAI 4616 2014-10-03 14:24:00.000 grid3.done/num_4600_4699/hmi.sharp_cea_720s.46... 7c3da7a4dbd08da4ccee60fecd7e0118 1635068160 792 1012 680 545022720 3 True 20220301 need check size 17.964899 4616201410031424003
73743 4698.20141024_160000_TAI 4698 2014-10-24 16:00:00.000 grid3.done/num_4600_4699/hmi.sharp_cea_720s.46... 851980aacc23901e5ff3a654debe013f 3024076800 800 1472 856 1008025600 3 True 20220301 need check size 16.359800 4698201410241600003
73744 4640.20141006_173600_TAI 4640 2014-10-06 17:36:00.000 grid3.done/num_4600_4699/hmi.sharp_cea_720s.46... e99415002ee27c14dcf7d08490a15840 1137168000 500 1192 636 379056000 3 True 20220301 need check size 23.332500 4640201410061736003
73745 5039.20150112_013600_TAI 5039 2015-01-12 01:36:00.000 grid3.done/num_5000_5099/hmi.sharp_cea_720s.50... d64654bd4ed69f201c34faf0b036b9c7 5301338112 1008 1712 1024 1767112704 3 True 20220301 need check size 17.475500 5039201501120136003
73746 5028.20150109_044800_TAI 5028 2015-01-09 04:48:00.000 grid3.done/num_5000_5099/hmi.sharp_cea_720s.50... 21f9ab2c5e376aec3f45cfbe487e3e23 1806686976 656 1268 724 602228992 3 True 20220301 need check size 22.137501 5028201501090448003

73747 rows × 16 columns

Use of label information¶

Read configuration¶

In [ ]:
import os
import json

# Read configuration file
nlfff_path_dataset_label="nlfff-dataset-flare-label"
config_path=os.path.join(nlfff_path_dataset_label,"config.json")
with open(config_path,"r") as f:
    config_label=json.loads(f.read())
In [ ]:
config_label
Out[ ]:
{'config_version': 1.3,
 'flare_info_path': 'label-202203/raw_file/flare_info_processed.csv',
 'prepare_file': {'knoaa_vflaretimelist_path': 'label-202203/prepare/knoaa_vflaretimelist.pickle',
  'ksharp_vnoaa_path': 'label-202203/prepare/ksharp_vnoaa.pickle'},
 'label_info_path': 'label-202203/process/label.csv'}

Label Information¶

In [ ]:
label_info_path=os.path.join(
    nlfff_path_dataset_label,
    config_label["label_info_path"]
)
label_info=pandas.read_csv(label_info_path)
In [ ]:
label_info
Out[ ]:
harpnum_trec harp_num t_rec_str now_flare_level now_flare_id h6_flare_level h6_flare_id h12_flare_level h12_flare_id h24_flare_level h24_flare_id h48_flare_level h48_flare_id t_rec_datetime h24_posmx h24_poscmx h48_posmx h48_poscmx h24_delta05 h48_delta05
0 1.20100504_160000_TAI 1 20100504_160000_TAI 0 -1 0 -1 0 -1 0 -1 0 -1 2010-05-04 16:00:00.000 0 0 0 0 0 0
1 1.20100504_173600_TAI 1 20100504_173600_TAI 0 -1 0 -1 0 -1 0 -1 0 -1 2010-05-04 17:36:00.000 0 0 0 0 0 0
2 1.20100504_191200_TAI 1 20100504_191200_TAI 0 -1 0 -1 0 -1 0 -1 0 -1 2010-05-04 19:12:00.000 0 0 0 0 0 0
3 1.20100504_204800_TAI 1 20100504_204800_TAI 0 -1 0 -1 0 -1 0 -1 0 -1 2010-05-04 20:48:00.000 0 0 0 0 0 0
4 1.20100504_222400_TAI 1 20100504_222400_TAI 0 -1 0 -1 0 -1 0 -1 0 -1 2010-05-04 22:24:00.000 0 0 0 0 0 0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
46109 997.20111030_080000_TAI 997 20111030_080000_TAI 0 -1 324 67189 324 67189 324 67189 324 67189 2011-10-30 08:00:00.000 0 1 0 1 3 3
46110 997.20111030_093600_TAI 997 20111030_093600_TAI 324 67189 324 67189 324 67189 324 67189 324 67189 2011-10-30 09:36:00.000 0 2 0 2 0 0
46111 997.20111030_111200_TAI 997 20111030_111200_TAI 0 -1 319 67192 319 67192 323 67197 323 67197 2011-10-30 11:12:00.000 0 1 0 1 3 3
46112 997.20111030_124800_TAI 997 20111030_124800_TAI 0 -1 319 67192 319 67192 323 67197 323 67197 2011-10-30 12:48:00.000 0 1 0 1 3 3
46113 997.20111030_142400_TAI 997 20111030_142400_TAI 0 -1 319 67192 323 67197 323 67197 323 67197 2011-10-30 14:24:00.000 0 1 0 1 3 3

46114 rows × 20 columns

Flare Information¶

Record the flare details, you can perform this step if you need the flare details.

In [ ]:
flare_info_path=os.path.join(
    nlfff_path_dataset_label,
    config_label["flare_info_path"]
)

flare_info=pandas.read_csv(flare_info_path)
In [ ]:
flare_info
Out[ ]:
deeps_flare_id start_datetime end_datetime peak_datetime xray_class xray_intensity latitude longtitude noaa_ar source
0 78524 2016-01-01 23:10:00.000 2016-01-02 01:01:00.000 2016-01-02 00:11:00.000 M 23.0 NaN NaN 12473.0 swpc
1 32340 1991-12-18 06:06:00.000 1991-12-18 06:39:00.000 1991-12-18 06:38:00.000 C 19.0 NaN NaN NaN goes
2 19726 1986-05-02 16:36:00.000 1986-05-02 16:44:00.000 1986-05-02 16:41:00.000 C 12.0 NaN NaN NaN goes
3 36328 1993-05-26 01:54:00.000 1993-05-26 02:03:00.000 1993-05-26 01:59:00.000 C 32.0 NaN NaN NaN goes
4 26657 1990-01-23 05:03:00.000 1990-01-23 05:35:00.000 1990-01-23 05:04:00.000 M 23.0 -10.0 56.0 5900.0 goes
... ... ... ... ... ... ... ... ... ... ...
85509 72167 2014-03-12 22:28:00.000 2014-03-12 22:39:00.000 2014-03-12 22:34:00.000 M 93.0 15.0 -78.0 11996.0 goes
85510 44281 1998-12-18 15:59:00.000 1998-12-18 16:13:00.000 1998-12-18 16:05:00.000 C 20.0 19.0 65.0 8415.0 goes
85511 30992 1991-07-20 06:51:00.000 1991-07-20 07:07:00.000 1991-07-20 06:54:00.000 C 34.0 -14.0 72.0 6737.0 goes
85512 76690 2016-08-09 01:55:00.000 2016-08-09 02:09:00.000 2016-08-09 01:58:00.000 B 76.0 NaN NaN NaN goes
85513 11311 1982-06-05 12:39:00.000 1982-06-05 12:46:00.000 1982-06-05 12:42:00.000 C 38.0 NaN NaN NaN goes

85514 rows × 10 columns

Using database archived information¶

In [ ]:
import json
import sqlite3
import os

Read configuration¶

In [ ]:
import os
import json

# Read configuration file
nlfff_path_dataset_archive="nlfff-dataset-and-flare-label-db-archive"
config_path=os.path.join(nlfff_path_dataset_archive,"config.json")
with open(config_path,"r") as f:
    config_db=json.loads(f.read())
In [ ]:
config_db
Out[ ]:
{'config_version': 1.3, 'archive_db_path': 'sqlite-202203/nlfff-202203'}
In [ ]:
archive_db_path=os.path.join(
    nlfff_path_dataset_archive,
    config_db["archive_db_path"]
)
archive_db_path
Out[ ]:
'nlfff-dataset-and-flare-label-db-archive/sqlite-202203/nlfff-202203'

Basic introduction to archived files¶

The archive file storage format is a sqlite database file, including the following information:

  • Product information of the sample (nx,ny,nz)
  • Sample label (CMX flare label)
  • flare information (start time)

Basic use¶

Query table and table information

In [ ]:
# Query which tables exist
query_sql="select name from sqlite_master where type='table' order by name; "
table_list=[]
with sqlite3.connect(archive_db_path) as conn:
   c = conn.cursor()
   cursor = c.execute(query_sql)
   for table in cursor:
      print(table)
      table_list.append(table[0])
print(table_list)

('flare_info',)
('nlfff_archive',)
('nlfff_flare_label',)
('nlfff_raw',)
['flare_info', 'nlfff_archive', 'nlfff_flare_label', 'nlfff_raw']
In [ ]:
# Query what columns are in a table
table_name="nlfff_archive"
query_sql="PRAGMA table_info({});".format(table_name)
column_list=[]
with sqlite3.connect(archive_db_path) as conn:
   c = conn.cursor()
   cursor = c.execute(query_sql)
   for column in cursor:
      print(column)
      column_list.append(column[1])
print(column_list)

(0, 'harpnum_trec', 'VARCHAR(50)', 0, None, 0)
(1, 'sharp_num', 'INTEGER', 0, None, 0)
(2, 't_rec', 'VARCHAR(50)', 0, None, 0)
(3, 'bout_path', 'VARCHAR(128)', 0, None, 0)
(4, 'bout_md5', 'VARCHAR(50)', 0, None, 0)
(5, 'bout_size', 'INTEGER', 0, None, 0)
(6, 'grid_x', 'INTEGER', 0, None, 0)
(7, 'grid_y', 'INTEGER', 0, None, 0)
(8, 'grid_z', 'INTEGER', 0, None, 0)
(9, 'grid_xyz', 'INTEGER', 0, None, 0)
(10, 'bout_maxlevel', 'INTEGER', 0, None, 0)
(11, 'bout_quality', 'BOOLEAN', 0, None, 0)
(12, 'batch', 'INTEGER', 0, None, 0)
(13, 'notes', 'VARCHAR(50)', 0, None, 0)
(14, 'bout_quality_value', 'REAL', 0, None, 0)
(15, 'identifiers', 'VARCHAR(50)', 0, None, 0)
['harpnum_trec', 'sharp_num', 't_rec', 'bout_path', 'bout_md5', 'bout_size', 'grid_x', 'grid_y', 'grid_z', 'grid_xyz', 'bout_maxlevel', 'bout_quality', 'batch', 'notes', 'bout_quality_value', 'identifiers']
In [ ]:
# Create a mapping dictionary of table names and column names

ktablename_vcolumnlist=dict()

for table_name in table_list:
   print(table_name)
   # table_name="nlfff_archive"
   query_sql="PRAGMA table_info({});".format(table_name)
   column_list=[]
   with sqlite3.connect(archive_db_path) as conn:
      c = conn.cursor()
      cursor = c.execute(query_sql)
      for column in cursor:
         # print(column)
         column_list.append(column[1])
   # print(column_list)
   ktablename_vcolumnlist[table_name]=column_list

flare_info
nlfff_archive
nlfff_flare_label
nlfff_raw
In [ ]:
ktablename_vcolumnlist
Out[ ]:
{'flare_info': ['deeps_flare_id',
  'start_datetime',
  'end_datetime',
  'peak_datetime',
  'xray_class',
  'xray_intensity',
  'latitude',
  'longtitude',
  'noaa_ar',
  'source'],
 'nlfff_archive': ['harpnum_trec',
  'sharp_num',
  't_rec',
  'bout_path',
  'bout_md5',
  'bout_size',
  'grid_x',
  'grid_y',
  'grid_z',
  'grid_xyz',
  'bout_maxlevel',
  'bout_quality',
  'batch',
  'notes',
  'bout_quality_value',
  'identifiers'],
 'nlfff_flare_label': ['harpnum_trec',
  'harp_num',
  't_rec_str',
  'now_flare_level',
  'now_flare_id',
  'h6_flare_level',
  'h6_flare_id',
  'h12_flare_level',
  'h12_flare_id',
  'h24_flare_level',
  'h24_flare_id',
  'h48_flare_level',
  'h48_flare_id',
  't_rec_datetime',
  'h24_posmx',
  'h24_poscmx',
  'h48_posmx',
  'h48_poscmx',
  'h24_delta05',
  'h48_delta05'],
 'nlfff_raw': ['harpnum_trec',
  'HARP_NUM',
  'T_REC_DATETIME',
  'T_REC_STR',
  'FILE_NAME',
  'query',
  'DATE',
  'DATE_S',
  'DATE_B',
  'DATE__OBS',
  'T_OBS',
  'T_REC',
  'T_REC_epoch',
  'T_REC_step',
  'T_REC_unit',
  'CADENCE',
  'USFLUX',
  'MEANGAM',
  'MEANGBT',
  'MEANGBZ',
  'MEANGBH',
  'MEANJZD',
  'TOTUSJZ',
  'MEANALP',
  'MEANJZH',
  'TOTUSJH',
  'ABSNJZH',
  'SAVNCPP',
  'MEANPOT',
  'TOTPOT',
  'MEANSHR',
  'SHRGT45',
  'R_VALUE',
  'GWILL',
  'CTYPE1',
  'CTYPE2',
  'CRPIX1',
  'CRPIX2',
  'CRVAL1',
  'CRVAL2',
  'CDELT1',
  'CDELT2',
  'CUNIT1',
  'CUNIT2',
  'IMCRPIX1',
  'IMCRPIX2',
  'IMCRVAL1',
  'IMCRVAL2',
  'CROTA2',
  'CRDER1',
  'CRDER2',
  'CSYSER1',
  'CSYSER2',
  'WCSNAME',
  'DSUN_OBS',
  'DSUN_REF',
  'RSUN_REF',
  'CRLN_OBS',
  'CRLT_OBS',
  'CAR_ROT',
  'OBS_VR',
  'OBS_VW',
  'OBS_VN',
  'RSUN_OBS',
  'TELESCOP',
  'INSTRUME',
  'WAVELNTH',
  'CAMERA',
  'QUALITY',
  'QUAL_S',
  'QUALLEV1',
  'BUNIT_000',
  'BUNIT_001',
  'BUNIT_002',
  'BUNIT_003',
  'BUNIT_004',
  'BUNIT_005',
  'BUNIT_006',
  'BUNIT_007',
  'BUNIT_008',
  'BUNIT_009',
  'BUNIT_010',
  'ORIGIN',
  'CONTENT',
  'HISTORY',
  'COMMENT',
  'BLD_VERS',
  'CALVER64',
  'CODEVER7',
  'HFLID',
  'HCFTID',
  'QLOOK',
  'HARPNUM',
  'MASK',
  'ARM_QUAL',
  'ARM_NCLN',
  'H_MERGE',
  'H_FAINT',
  'ARM_MODL',
  'ARM_EDGE',
  'ARM_BETA',
  'LATDTMIN',
  'LONDTMIN',
  'LATDTMAX',
  'LONDTMAX',
  'OMEGA_DT',
  'NPIX',
  'SIZE',
  'AREA',
  'NACR',
  'SIZE_ACR',
  'AREA_ACR',
  'MTOT',
  'MNET',
  'MPOS_TOT',
  'MNEG_TOT',
  'MMEAN',
  'MSTDEV',
  'MSKEW',
  'MKURT',
  'LAT_MIN',
  'LON_MIN',
  'LAT_MAX',
  'LON_MAX',
  'LAT_FWT',
  'LON_FWT',
  'LAT_FWTPOS',
  'LON_FWTPOS',
  'LAT_FWTNEG',
  'LON_FWTNEG',
  'T_FRST',
  'T_FRST1',
  'T_LAST1',
  'T_LAST',
  'N_PATCH',
  'N_PATCH1',
  'N_PATCHM',
  'NOAA_AR',
  'NOAA_NUM',
  'NOAA_ARS',
  'INVCODEV',
  'INVDOCU',
  'INVITERA',
  'INVSTLGT',
  'INVFLPRF',
  'INVPHMAP',
  'INVVLAVE',
  'INVBLAVE',
  'INVBBAVE',
  'INVNPRCS',
  'INVNCNVG',
  'AMBCODEV',
  'AMBDOCU',
  'AMBGMTRY',
  'AMBPATCH',
  'AMBWEAK',
  'AMBNEROD',
  'AMBNGROW',
  'AMBNPAD',
  'AMBNAP',
  'AMBNTX',
  'AMBNTY',
  'AMBBTHR0',
  'AMBBTHR1',
  'AMBSEED',
  'AMBNEQ',
  'AMBLMBDA',
  'AMBTFCT0',
  'AMBTFCTR',
  'DATAVALS_000',
  'DATAVALS_001',
  'DATAVALS_002',
  'DATAVALS_003',
  'DATAVALS_004',
  'DATAVALS_005',
  'DATAVALS_006',
  'DATAVALS_007',
  'DATAVALS_008',
  'DATAVALS_009',
  'DATAVALS_010',
  'MISSVALS_000',
  'MISSVALS_001',
  'MISSVALS_002',
  'MISSVALS_003',
  'MISSVALS_004',
  'MISSVALS_005',
  'MISSVALS_006',
  'MISSVALS_007',
  'MISSVALS_008',
  'MISSVALS_009',
  'MISSVALS_010',
  'DATAMIN_001',
  'DATAMIN_002',
  'DATAMIN_003',
  'DATAMIN_004',
  'DATAMIN_005',
  'DATAMIN_006',
  'DATAMIN_007',
  'DATAMIN_008',
  'DATAMIN_009',
  'DATAMIN_010',
  'DATAMIN_000',
  'DATAMAX_000',
  'DATAMAX_001',
  'DATAMAX_002',
  'DATAMAX_003',
  'DATAMAX_004',
  'DATAMAX_005',
  'DATAMAX_006',
  'DATAMAX_007',
  'DATAMAX_008',
  'DATAMAX_009',
  'DATAMAX_010',
  'DATAMEDN_003',
  'DATAMEDN_004',
  'DATAMEDN_005',
  'DATAMEDN_006',
  'DATAMEDN_007',
  'DATAMEDN_008',
  'DATAMEDN_009',
  'DATAMEDN_010',
  'DATAMEDN_000',
  'DATAMEDN_001',
  'DATAMEDN_002',
  'DATAMEAN_010',
  'DATAMEAN_000',
  'DATAMEAN_001',
  'DATAMEAN_002',
  'DATAMEAN_003',
  'DATAMEAN_004',
  'DATAMEAN_005',
  'DATAMEAN_006',
  'DATAMEAN_007',
  'DATAMEAN_008',
  'DATAMEAN_009',
  'DATARMS_000',
  'DATARMS_001',
  'DATARMS_002',
  'DATARMS_003',
  'DATARMS_004',
  'DATARMS_005',
  'DATARMS_006',
  'DATARMS_007',
  'DATARMS_008',
  'DATARMS_009',
  'DATARMS_010',
  'ERRGAM',
  'ERRTAI',
  'ERRBH',
  'ERRMPOT',
  'ERRBT',
  'ERRTUI',
  'ERRBZ',
  'CMASK',
  'ERRJZ',
  'ERRVF',
  'ERRALP',
  'ERRMIH',
  'ERRMSHA',
  'ERRUSI',
  'DOFFSET',
  'ERRTPOT',
  'ERRJHT',
  'magnetogram',
  'bitmap',
  'Dopplergram',
  'continuum',
  'Bp',
  'Bt',
  'Br',
  'Bp_err',
  'Bt_err',
  'Br_err',
  'conf_disambig',
  'BHARP',
  'MHARP']}

Query Table Data¶

In [ ]:
# Sample Raw Fits File Header File Information(time,noaa number)

table_name="nlfff_raw"
with sqlite3.connect(archive_db_path) as conn:
   c = conn.cursor()

   query_sql="SELECT *  from {}".format(table_name)
   cursor = c.execute(query_sql)
   for row in cursor:
      row_data_list=list(row)
      row_column_name_list=ktablename_vcolumnlist[table_name]
      row_dict=dict(zip(row_column_name_list,row_data_list))
      print(row_dict)
      break

{'harpnum_trec': '1644.20120516_031200_TAI', 'HARP_NUM': 1644, 'T_REC_DATETIME': '2012-05-16 03:12:00.000', 'T_REC_STR': '20120516_031200_TAI', 'FILE_NAME': 'hmi.sharp_cea_720s.1644.20120516_031200_TAI', 'query': 'hmi.sharp_cea_720s[1644][2012.05.16_03:12:00_TAI]', 'DATE': '2018-05-09T03:05:47Z', 'DATE_S': '2012-05-20T19:22:46Z', 'DATE_B': '2015-09-30T20:47:04Z', 'DATE__OBS': '2012-05-16T03:10:24.10Z', 'T_OBS': '2012.05.16_03:12:06_TAI', 'T_REC': '2012.05.16_03:12:00_TAI', 'T_REC_epoch': '1993.01.01_00:00:00_TAI', 'T_REC_step': 720.0, 'T_REC_unit': 'secs', 'CADENCE': 720.0, 'USFLUX': 7.195475e+21, 'MEANGAM': 47.02, 'MEANGBT': 113.15, 'MEANGBZ': 110.28299999999999, 'MEANGBH': 61.652, 'MEANJZD': 0.12072397, 'TOTUSJZ': 9593281000000, 'MEANALP': 0.01153706, 'MEANJZH': 0.0052498, 'TOTUSJH': 474.784, 'ABSNJZH': 57.433, 'SAVNCPP': 710574700000, 'MEANPOT': 9327.876, 'TOTPOT': 1.355353e+23, 'MEANSHR': 36.965, 'SHRGT45': 33.726, 'R_VALUE': 2.103, 'GWILL': '', 'CTYPE1': 'CRLN-CEA', 'CTYPE2': 'CRLT-CEA', 'CRPIX1': 477.5, 'CRPIX2': 279.0, 'CRVAL1': 134.58140600000002, 'CRVAL2': -22.961349, 'CDELT1': 0.03, 'CDELT2': 0.03, 'CUNIT1': 'degree', 'CUNIT2': 'degree', 'IMCRPIX1': 2035.023315, 'IMCRPIX2': 2052.667725, 'IMCRVAL1': 0.0, 'IMCRVAL2': 0.0, 'CROTA2': 0.0, 'CRDER1': '', 'CRDER2': '', 'CSYSER1': '', 'CSYSER2': '', 'WCSNAME': 'Carrington Heliographic', 'DSUN_OBS': 151288617876.01, 'DSUN_REF': 149597870691, 'RSUN_REF': 696000000, 'CRLN_OBS': 122.84070600000001, 'CRLT_OBS': -2.505391, 'CAR_ROT': 2123, 'OBS_VR': 1939.0652760000003, 'OBS_VW': 30084.039826, 'OBS_VN': 6001.302302, 'RSUN_OBS': 948.920105, 'TELESCOP': 'SDO/HMI', 'INSTRUME': 'HMI_SIDE1', 'WAVELNTH': 6173.0, 'CAMERA': 1, 'QUALITY': '0x00000000', 'QUAL_S': '0x00000000', 'QUALLEV1': '0x00000000', 'BUNIT_000': 'Mx/cm^2', 'BUNIT_001': ' ', 'BUNIT_002': 'cm/s', 'BUNIT_003': 'DN/s', 'BUNIT_004': 'Mx/cm^2', 'BUNIT_005': 'Mx/cm^2', 'BUNIT_006': 'Mx/cm^2', 'BUNIT_007': 'Mx/cm^2', 'BUNIT_008': 'Mx/cm^2', 'BUNIT_009': 'Mx/cm^2', 'BUNIT_010': ' ', 'ORIGIN': 'SDO/JSOC-SDP', 'CONTENT': 'HMI observable', 'HISTORY': ' ', 'COMMENT': ' ', 'BLD_VERS': 'V8R9X', 'CALVER64': '0x0000000000000010', 'CODEVER7': '$Id: sharp.c,v 1.38 2015/03/18 00:28:26 xudong Exp $', 'HFLID': 1021, 'HCFTID': 11, 'QLOOK': 0, 'HARPNUM': 1644, 'MASK': 32, 'ARM_QUAL': '0x00000000', 'ARM_NCLN': 1, 'H_MERGE': 0, 'H_FAINT': 0, 'ARM_MODL': '/builtin/hmi.M_Ic_noLimbDark_720s.production', 'ARM_EDGE': 2.5, 'ARM_BETA': 0.3, 'LATDTMIN': -31.313299, 'LONDTMIN': -2.5699, 'LATDTMAX': -14.6094, 'LONDTMAX': 26.0513, 'OMEGA_DT': 13.2649, 'NPIX': 133203, 'SIZE': 11976.625, 'AREA': 6635.683594, 'NACR': 9287, 'SIZE_ACR': 835.018066, 'AREA_ACR': 462.282288, 'MTOT': 6930884.5, 'MNET': 260587.09375, 'MPOS_TOT': 3595736.0, 'MNEG_TOT': 3335148.75, 'MMEAN': 1.956316, 'MSTDEV': 144.275909, 'MSKEW': 4.339589, 'MKURT': 41.763123, 'LAT_MIN': -30.760214, 'LON_MIN': -1.510393, 'LAT_MAX': -15.120847, 'LON_MAX': 23.849808, 'LAT_FWT': -23.853493, 'LON_FWT': 12.266521000000001, 'LAT_FWTPOS': -23.781185, 'LON_FWTPOS': 14.561793, 'LAT_FWTNEG': -23.969935999999997, 'LON_FWTNEG': 8.570277, 'T_FRST': '2012.05.06_21:12:00_TAI', 'T_FRST1': '2012.05.07_21:12:00_TAI', 'T_LAST1': '2012.05.22_00:48:00_TAI', 'T_LAST': '2012.05.23_00:48:00_TAI', 'N_PATCH': 1939, 'N_PATCH1': 1699, 'N_PATCHM': 0, 'NOAA_AR': 11477, 'NOAA_NUM': 2, 'NOAA_ARS': '11477,11478', 'INVCODEV': 'vfisv FD10 2012 Apr. 13', 'INVDOCU': '', 'INVITERA': 200, 'INVSTLGT': 'No', 'INVFLPRF': ' ', 'INVPHMAP': ' ', 'INVVLAVE': 1.033407e+22, 'INVBLAVE': 20.044, 'INVBBAVE': 127.6648, 'INVNPRCS': 150736, 'INVNCNVG': 11121934, 'AMBCODEV': 'disambig_v3 2013 Dec 06', 'AMBDOCU': '', 'AMBGMTRY': 2, 'AMBPATCH': 0, 'AMBWEAK': 2, 'AMBNEROD': 1, 'AMBNGROW': 5, 'AMBNPAD': 200, 'AMBNAP': 10, 'AMBNTX': 30, 'AMBNTY': 30, 'AMBBTHR0': 200.0, 'AMBBTHR1': 400.0, 'AMBSEED': 4, 'AMBNEQ': 100, 'AMBLMBDA': 1.0, 'AMBTFCT0': 2.0, 'AMBTFCTR': 0.98, 'DATAVALS_000': 531378, 'DATAVALS_001': 531378, 'DATAVALS_002': 531378, 'DATAVALS_003': 531378, 'DATAVALS_004': 531378, 'DATAVALS_005': 531378, 'DATAVALS_006': 531378, 'DATAVALS_007': 531378, 'DATAVALS_008': 531378, 'DATAVALS_009': 531378, 'DATAVALS_010': 531378, 'MISSVALS_000': 0, 'MISSVALS_001': 0, 'MISSVALS_002': 0, 'MISSVALS_003': 0, 'MISSVALS_004': 0, 'MISSVALS_005': 0, 'MISSVALS_006': 0, 'MISSVALS_007': 0, 'MISSVALS_008': 0, 'MISSVALS_009': 0, 'MISSVALS_010': 0, 'DATAMIN_001': 782.0, 'DATAMIN_002': 7730.0, 'DATAMIN_003': -1080.859985, 'DATAMIN_004': -1407.140015, 'DATAMIN_005': -1220.349976, 'DATAMIN_006': 5.495, 'DATAMIN_007': 5.89, 'DATAMIN_008': 3.056, 'DATAMIN_009': 50.0, 'DATAMIN_010': -883.099976, 'DATAMIN_000': 0.0, 'DATAMAX_000': 1839.199951, 'DATAMAX_001': 34.0, 'DATAMAX_002': 3945.0, 'DATAMAX_003': 64898.0, 'DATAMAX_004': 1343.079956, 'DATAMAX_005': 1352.119995, 'DATAMAX_006': 2752.580078, 'DATAMAX_007': 1515.130005, 'DATAMAX_008': 639.731018, 'DATAMAX_009': 361.239014, 'DATAMAX_010': 90.0, 'DATAMEDN_003': -4.422134, 'DATAMEDN_004': 41.280186, 'DATAMEDN_005': 40.248523999999996, 'DATAMEDN_006': 18.449409, 'DATAMEDN_007': 50.000305, 'DATAMEDN_008': -0.7078, 'DATAMEDN_009': 0.999985, 'DATAMEDN_010': 2272.499023, 'DATAMEDN_000': 56360.22265599999, 'DATAMEDN_001': 4.672016999999999, 'DATAMEDN_002': -9.005756, 'DATAMEAN_010': 20.131287, 'DATAMEAN_000': 52.015175, 'DATAMEAN_001': 0.475767, 'DATAMEAN_002': 10.057181, 'DATAMEAN_003': 2277.6877440000003, 'DATAMEAN_004': 56307.226562, 'DATAMEAN_005': 5.432481, 'DATAMEAN_006': -6.025362, 'DATAMEAN_007': 1.694795, 'DATAMEAN_008': 44.34304, 'DATAMEAN_009': 43.243214, 'DATARMS_000': 80.526901, 'DATARMS_001': 14.483751000000002, 'DATARMS_002': 386.348206, 'DATARMS_003': 2445.601807, 'DATARMS_004': 77.775124, 'DATARMS_005': 82.911629, 'DATARMS_006': 115.430428, 'DATARMS_007': 16.401438, 'DATARMS_008': 15.701928, 'DATARMS_009': 9.27594, 'DATARMS_010': 6.639278999999999, 'ERRGAM': 0.033, 'ERRTAI': 7.831, 'ERRBH': 0.237, 'ERRMPOT': 0.1888691, 'ERRBT': 10.901, 'ERRTUI': 0.16, 'ERRBZ': 7.831, 'CMASK': 10941.0, 'ERRJZ': 3261369466163822100, 'ERRVF': 0.07241935, 'ERRALP': 0.00157312, 'ERRMIH': 0.00071585, 'ERRMSHA': 0.003, 'ERRUSI': 105226300000, 'DOFFSET': 1054507008, 'ERRTPOT': 1.58388e+20, 'ERRJHT': 50.0, 'magnetogram': '/SUM90/D1042885471/S00000/magnetogram.fits', 'bitmap': '/SUM90/D1042885471/S00000/bitmap.fits', 'Dopplergram': '/SUM90/D1042885471/S00000/Dopplergram.fits', 'continuum': '/SUM90/D1042885471/S00000/continuum.fits', 'Bp': '/SUM90/D1042885471/S00000/Bp.fits', 'Bt': '/SUM90/D1042885471/S00000/Bt.fits', 'Br': '/SUM90/D1042885471/S00000/Br.fits', 'Bp_err': '/SUM90/D1042885471/S00000/Bp_err.fits', 'Bt_err': '/SUM90/D1042885471/S00000/Bt_err.fits', 'Br_err': '/SUM90/D1042885471/S00000/Br_err.fits', 'conf_disambig': '/SUM90/D1042885471/S00000/conf_disambig.fits', 'BHARP': 'InvalidLink', 'MHARP': 'hmi.Mharp_720s[1644][2012.05.16_03:12:00_TAI]'}
In [ ]:
# Product information for samples (nx,ny,nz）

table_name="nlfff_archive"
with sqlite3.connect(archive_db_path) as conn:
   c = conn.cursor()

   query_sql="SELECT *  from {}".format(table_name)
   cursor = c.execute(query_sql)
   for row in cursor:
      row_data_list=list(row)
      row_column_name_list=ktablename_vcolumnlist[table_name]
      row_dict=dict(zip(row_column_name_list,row_data_list))
      print(row_dict)
      break

{'harpnum_trec': '6558.20160520_044800_TAI', 'sharp_num': 6558, 't_rec': '2016-05-20 04:48:00.000', 'bout_path': 'grid3.done/num_6500_6599/hmi.sharp_cea_720s.6558.20160520_044800_TAI/Bout.bin', 'bout_md5': 'ea4c960822a1284fe43637b1717bc30b', 'bout_size': 3580360704, 'grid_x': 432, 'grid_y': 764, 'grid_z': 452, 'grid_xyz': 149181696, 'bout_maxlevel': 3, 'bout_quality': 'true', 'batch': 20220301, 'notes': 'new', 'bout_quality_value': 15.568, 'identifiers': '6558201605200448003'}
In [ ]:
# Sample labels (CMX flare label)

table_name="nlfff_flare_label"
with sqlite3.connect(archive_db_path) as conn:
   c = conn.cursor()

   query_sql="SELECT *  from {}".format(table_name)
   cursor = c.execute(query_sql)
   for row in cursor:
      row_data_list=list(row)
      row_column_name_list=ktablename_vcolumnlist[table_name]
      row_dict=dict(zip(row_column_name_list,row_data_list))
      print(row_dict)
      break

{'harpnum_trec': '1021.20111031_093600_TAI', 'harp_num': 1021, 't_rec_str': '20111031_093600_TAI', 'now_flare_level': 0, 'now_flare_id': -1, 'h6_flare_level': 0, 'h6_flare_id': -1, 'h12_flare_level': 0, 'h12_flare_id': -1, 'h24_flare_level': 0, 'h24_flare_id': -1, 'h48_flare_level': 0, 'h48_flare_id': -1, 't_rec_datetime': '2011-10-31 09:36:00.000', 'h24_posmx': 0, 'h24_poscmx': 0, 'h48_posmx': 0, 'h48_poscmx': 0, 'h24_delta05': 0, 'h48_delta05': 0}
In [ ]:
# Flare information (start time)

table_name="flare_info"
with sqlite3.connect(archive_db_path) as conn:
   c = conn.cursor()

   query_sql="SELECT *  from {}".format(table_name)
   cursor = c.execute(query_sql)
   for row in cursor:
      row_data_list=list(row)
      row_column_name_list=ktablename_vcolumnlist[table_name]
      row_dict=dict(zip(row_column_name_list,row_data_list))
      print(row_dict)
      break

{'deeps_flare_id': 78524, 'start_datetime': '2016-01-01 23:10:00.000', 'end_datetime': '2016-01-02 01:01:00.000', 'peak_datetime': '2016-01-02 00:11:00.000', 'xray_class': 'M', 'xray_intensity': 23, 'latitude': '', 'longtitude': '', 'noaa_ar': 12473, 'source': 'swpc'}

Advanced queries¶

  • python scripts can use pandas + an engine such as pandas+sqlalchemy
  • visualization programs such as dbeaver
  • other language toolkit links sqlite

Using a single sample (basic use)¶

Basic reading¶

Get the storage path and time¶

The following information is required at a minimum to use a sample:

  • nonlinear force-free field data Bout.bin

  • Bout.bin corresponding to nx, ny, nz

  • the harp number and time of the sample

In [ ]:
# Inside the dataset summary information there is greater than or equal to one sample, 
# We use this sample as an example for illustration 

# Take sample 0 as an example and read the `harpnum_trec` and path  of the already downloaded sample
nlfff_path_dataset_summary="nlfff-dataset-info"
sample_num=0
sample_harpnum_trec=config_ds["demo_sample"][sample_num]["harpnum_trec"]
sample_path=os.path.join(
    nlfff_path_dataset_summary,
    config_ds["demo_sample"][sample_num]["path"])
In [ ]:
# Harp number and time of the sample 
sample_harpnum_trec
Out[ ]:
'7327.20181220_093600_TAI'
In [ ]:
# Bout path
sample_bout_path=os.path.join(sample_path,"Bout.bin")
sample_bout_path
Out[ ]:
'nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/Bout.bin'

Get sample product file information (nx ny nz)¶

In [ ]:
# Get sample product file information 

sample_product_info=product_info[ product_info["harpnum_trec"]==sample_harpnum_trec ]
In [ ]:
sample_product_info
Out[ ]:
harpnum_trec sharp_num t_rec bout_path bout_md5 bout_size grid_x grid_y grid_z grid_xyz bout_maxlevel bout_quality batch notes bout_quality_value identifiers
49935 7327.20181220_093600_TAI 7327 2018-12-20 09:36:00.000 grid3.done/num_7300_7399/hmi.sharp_cea_720s.73... f7487ece2a486ba1aba7943398022fbf 107980800 100 296 152 4499200 3 True 20220301 NaN 18.8298 7327201812200936003
In [ ]:
# Maximum calculation level of the sample 
sample_bout_maxlevel=int(sample_product_info["bout_maxlevel"])
# The nx, ny, nz of the sample are 
sample_nx=int(sample_product_info["grid_x"])
sample_ny=int(sample_product_info["grid_y"])
sample_nz=int(sample_product_info["grid_z"])
# The identifiers of the sample are
sample_identifiers=int(sample_product_info["identifiers"])

Read data¶

In [ ]:
# Read data according to nx ny nz 
import numpy

nx=sample_nx
ny=sample_ny
nz=sample_nz

np_dtype_str=r"<d"
bin_path=os.path.join(sample_path,"Bout.bin")

# https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
nlfff_data = numpy.memmap(sample_bout_path,
                dtype=numpy.dtype(np_dtype_str),
                offset=0,
                shape=(3, nx, ny, nz),
                order='C')
In [ ]:
nlfff_data.shape
Out[ ]:
(3, 100, 296, 152)

Get association information¶

Get sample raw header file information¶

In [ ]:
# Get sample raw header file information 

sample_raw_info=raw_info[ raw_info["harpnum_trec"]==sample_harpnum_trec ]
In [ ]:
sample_raw_info
Out[ ]:
harpnum_trec HARP_NUM T_REC_DATETIME T_REC_STR FILE_NAME query DATE DATE_S DATE_B DATE__OBS ... continuum Bp Bt Br Bp_err Bt_err Br_err conf_disambig BHARP MHARP
33038 7327.20181220_093600_TAI 7327 2018-12-20 09:36:00.000 20181220_093600_TAI hmi.sharp_cea_720s.7327.20181220_093600_TAI hmi.sharp_cea_720s[7327][2018.12.20_09:36:00_TAI] 2019-01-25T04:03:11Z 2018-12-24T16:25:13Z 2018-12-26T00:57:23Z 2018-12-20T09:34:30.00Z ... /SUM34/D1129547914/S00000/continuum.fits /SUM34/D1129547914/S00000/Bp.fits /SUM34/D1129547914/S00000/Bt.fits /SUM34/D1129547914/S00000/Br.fits /SUM34/D1129547914/S00000/Bp_err.fits /SUM34/D1129547914/S00000/Bt_err.fits /SUM34/D1129547914/S00000/Br_err.fits /SUM34/D1129547914/S00000/conf_disambig.fits InvalidLink hmi.Mharp_720s[7327][2018.12.20_09:36:00_TAI]

1 rows × 276 columns

Get label information¶

In [ ]:
# Get the label of a sample 

# sample_harpnum_trec="997.20111030_142400_TAI"
sample_label=label_info[label_info["harpnum_trec"]==sample_harpnum_trec]
In [ ]:
# All label information for this sample 
sample_label
Out[ ]:
harpnum_trec harp_num t_rec_str now_flare_level now_flare_id h6_flare_level h6_flare_id h12_flare_level h12_flare_id h24_flare_level h24_flare_id h48_flare_level h48_flare_id t_rec_datetime h24_posmx h24_poscmx h48_posmx h48_poscmx h24_delta05 h48_delta05
43896 7327.20181220_093600_TAI 7327 20181220_093600_TAI 0 -1 0 -1 0 -1 0 -1 0 -1 2018-12-20 09:36:00.000 0 0 0 0 0 0

Get secondary association information¶

Label corresponding to flare information¶

In [ ]:
# Get a sample of positive samples with flares 

sample_harpnum_trec_pos="997.20111030_142400_TAI"
sample_label_pos=label_info[label_info["harpnum_trec"]==sample_harpnum_trec_pos]
In [ ]:
# All label information for this sample 
sample_label_pos
Out[ ]:
harpnum_trec harp_num t_rec_str now_flare_level now_flare_id h6_flare_level h6_flare_id h12_flare_level h12_flare_id h24_flare_level h24_flare_id h48_flare_level h48_flare_id t_rec_datetime h24_posmx h24_poscmx h48_posmx h48_poscmx h24_delta05 h48_delta05
46113 997.20111030_142400_TAI 997 20111030_142400_TAI 0 -1 319 67192 323 67197 323 67197 323 67197 2011-10-30 14:24:00.000 0 1 0 1 3 3
In [ ]:
# Further information can also be obtained on the flare information 

# Take the above sample with the information of flare occurrence within 12 hours as an example 
# Get the id of the flares that occurred in the above sample within 12 hours 
sample_h12_flare_id=int(sample_label_pos["h12_flare_id"])
In [ ]:
sample_h12_flare_id
Out[ ]:
67197
In [ ]:
# Corresponding flare details 
sample_h12_flare_detail_info=flare_info[flare_info["deeps_flare_id"]==sample_h12_flare_id]
In [ ]:
sample_h12_flare_detail_info
Out[ ]:
deeps_flare_id start_datetime end_datetime peak_datetime xray_class xray_intensity latitude longtitude noaa_ar source
71616 67197 2011-10-31 00:50:00.000 2011-10-31 01:29:00.000 2011-10-31 01:10:00.000 C 23.0 NaN NaN 11330.0 goes

Using a single sample (pynlfff toolkit)¶

Toolkit initialization¶

Install pynlfff toolkit¶

In [ ]:
# Install the latest toolkit 
!pip uninstall pynlfff -y
!pip install pynlfff

# Import Toolkit 
import pynlfff

Found existing installation: pynlfff 0.3.3.5
Uninstalling pynlfff-0.3.3.5:
  Successfully uninstalled pynlfff-0.3.3.5
Collecting pynlfff
  Using cached pynlfff-0.3.3.5-py3-none-any.whl (44 kB)
Requirement already satisfied: numpy in /home/zander/anaconda3/lib/python3.9/site-packages (from pynlfff) (1.20.3)
Requirement already satisfied: h5py in /home/zander/anaconda3/lib/python3.9/site-packages (from pynlfff) (3.3.0)
Requirement already satisfied: matplotlib in /home/zander/anaconda3/lib/python3.9/site-packages (from pynlfff) (3.4.3)
Requirement already satisfied: python-dateutil>=2.7 in /home/zander/anaconda3/lib/python3.9/site-packages (from matplotlib->pynlfff) (2.8.2)
Requirement already satisfied: pyparsing>=2.2.1 in /home/zander/anaconda3/lib/python3.9/site-packages (from matplotlib->pynlfff) (3.0.4)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/zander/anaconda3/lib/python3.9/site-packages (from matplotlib->pynlfff) (1.3.1)
Requirement already satisfied: pillow>=6.2.0 in /home/zander/anaconda3/lib/python3.9/site-packages (from matplotlib->pynlfff) (8.4.0)
Requirement already satisfied: cycler>=0.10 in /home/zander/anaconda3/lib/python3.9/site-packages (from matplotlib->pynlfff) (0.10.0)
Requirement already satisfied: six in /home/zander/anaconda3/lib/python3.9/site-packages (from cycler>=0.10->matplotlib->pynlfff) (1.16.0)
Installing collected packages: pynlfff
Successfully installed pynlfff-0.3.3.5

Install toolkit dependencies¶

In [ ]:
# Because the whole toolkit has more dependencies, 
# there is no setting to install all when downloading the toolkit

# You can use the following command to follow the toolkit as needed 
# Create object 
checker=pynlfff.RCheck()
# See which modules have dependencies to install 
checker.h()

This module is to test some module, for pynlfff which function you need, is install or not.
    Can use RCheck().check() or RCheck().check(0) for Full module test
    RCheck().check(1) for Base function module test 
    RCheck().check(2) for Prepare function module test 
    RCheck().check(3) for Computer function module test 
    RCheck().check(4) for Product function module test 
    RCheck().check(5) for Plot function module test 
    RCheck().check(6) for Download fits raw data function module test 
    RCheck().check(7) for Label function module test
In [ ]:
# Use 4 to install Product-related dependencies 
checker.check(4)

# Use 0 i.e. follow all dependencies 
checker.check(0)

Start Test
<module 'h5py' from '/home/zander/anaconda3/lib/python3.9/site-packages/h5py/__init__.py'>
True
<module 'numpy' from '/home/zander/anaconda3/lib/python3.9/site-packages/numpy/__init__.py'>
True
Test Finish, Result: [['h5py', 'Exists'], ['numpy', 'Exists']]
['h5py', 'Exists']
['numpy', 'Exists']
Start Test
<module 'pandas' from '/home/zander/anaconda3/lib/python3.9/site-packages/pandas/__init__.py'>
True
<module 'astropy' from '/home/zander/anaconda3/lib/python3.9/site-packages/astropy/__init__.py'>
True
<module 'numpy' from '/home/zander/anaconda3/lib/python3.9/site-packages/numpy/__init__.py'>
True
<module 'sunpy' from '/home/zander/anaconda3/lib/python3.9/site-packages/sunpy/__init__.py'>
True
<module 'drms' from '/home/zander/anaconda3/lib/python3.9/site-packages/drms/__init__.py'>
True
<module 'h5py' from '/home/zander/anaconda3/lib/python3.9/site-packages/h5py/__init__.py'>
True
<module 'matplotlib' from '/home/zander/anaconda3/lib/python3.9/site-packages/matplotlib/__init__.py'>
True
<module 'psutil' from '/home/zander/anaconda3/lib/python3.9/site-packages/psutil/__init__.py'>
True
<module 'requests' from '/home/zander/anaconda3/lib/python3.9/site-packages/requests/__init__.py'>
True
<module 'bs4' from '/home/zander/anaconda3/lib/python3.9/site-packages/bs4/__init__.py'>
True
<module 'lxml' from '/home/zander/anaconda3/lib/python3.9/site-packages/lxml/__init__.py'>
True
<module 'zeep' from '/home/zander/anaconda3/lib/python3.9/site-packages/zeep/__init__.py'>
True
<module 'wget' from '/home/zander/anaconda3/lib/python3.9/site-packages/wget.py'>
True
<module 'peewee' from '/home/zander/anaconda3/lib/python3.9/site-packages/peewee.py'>
True
<module 'vtk' from '/home/zander/anaconda3/lib/python3.9/site-packages/vtk.py'>
True
Test Finish, Result: [['pandas', 'Exists'], ['astropy', 'Exists'], ['numpy', 'Exists'], ['sunpy', 'Exists'], ['drms', 'Exists'], ['h5py', 'Exists'], ['matplotlib', 'Exists'], ['psutil', 'Exists'], ['requests', 'Exists'], ['bs4', 'Exists'], ['lxml', 'Exists'], ['zeep', 'Exists'], ['wget', 'Exists'], ['peewee', 'Exists'], ['vtk', 'Exists']]
['pandas', 'Exists']
['astropy', 'Exists']
['numpy', 'Exists']
['sunpy', 'Exists']
['drms', 'Exists']
['h5py', 'Exists']
['matplotlib', 'Exists']
['psutil', 'Exists']
['requests', 'Exists']
['bs4', 'Exists']
['lxml', 'Exists']
['zeep', 'Exists']
['wget', 'Exists']
['peewee', 'Exists']
['vtk', 'Exists']
Out[ ]:
[['pandas', 'Exists'],
 ['astropy', 'Exists'],
 ['numpy', 'Exists'],
 ['sunpy', 'Exists'],
 ['drms', 'Exists'],
 ['h5py', 'Exists'],
 ['matplotlib', 'Exists'],
 ['psutil', 'Exists'],
 ['requests', 'Exists'],
 ['bs4', 'Exists'],
 ['lxml', 'Exists'],
 ['zeep', 'Exists'],
 ['wget', 'Exists'],
 ['peewee', 'Exists'],
 ['vtk', 'Exists']]

Data Reading¶

Get the storage path and time¶

The following information is required at a minimum to use a sample:

  • nonlinear force-free field data Bout.bin

  • Bout.bin corresponding to nx, ny, nz

  • the harp number and time of the sample

In [ ]:
# Inside the dataset summary information there is greater than or equal to one sample, 
# We use this sample as an example for illustration 

# Take sample 0 as an example and read the `harpnum_trec` and path  of the already downloaded sample
nlfff_path_dataset_summary="nlfff-dataset-info"
sample_num=0
sample_harpnum_trec=config_ds["demo_sample"][sample_num]["harpnum_trec"]
sample_path=os.path.join(
    nlfff_path_dataset_summary,
    config_ds["demo_sample"][sample_num]["path"])
In [ ]:
# Harp number and time of the sample 
sample_harpnum_trec
Out[ ]:
'7327.20181220_093600_TAI'
In [ ]:
# Bout path
sample_bout_path=os.path.join(sample_path,"Bout.bin")
sample_bout_path
Out[ ]:
'nlfff-dataset-info/demo/hmi.sharp_cea_720s.7327.20181220_093600_TAI/Bout.bin'

Get sample product file information (nx ny nz)¶

In [ ]:
# Get sample product file information 

sample_product_info=product_info[ product_info["harpnum_trec"]==sample_harpnum_trec ]
sample_product_info
Out[ ]:
harpnum_trec sharp_num t_rec bout_path bout_md5 bout_size grid_x grid_y grid_z grid_xyz bout_maxlevel bout_quality batch notes bout_quality_value identifiers
49935 7327.20181220_093600_TAI 7327 2018-12-20 09:36:00.000 grid3.done/num_7300_7399/hmi.sharp_cea_720s.73... f7487ece2a486ba1aba7943398022fbf 107980800 100 296 152 4499200 3 True 20220301 NaN 18.8298 7327201812200936003
In [ ]:
# Maximum calculation level of the sample 
sample_bout_maxlevel=int(sample_product_info["bout_maxlevel"])
# The nx, ny, nz of the sample are
sample_nx=int(sample_product_info["grid_x"])
sample_ny=int(sample_product_info["grid_y"])
sample_nz=int(sample_product_info["grid_z"])
# The identifiers of the sample are
sample_identifiers=int(sample_product_info["identifiers"])

Read data¶

In [ ]:
from pynlfff.pyproduct import file

# Creating read objects
r=file.NlfffFile()
In [ ]:
r = file.NlfffFile()


bout_data = r.read_bin(sample_bout_path, nx=sample_nx,ny=sample_ny,nz=sample_nz)
print(bout_data.shape)
# print(s)

(3, 100, 296, 152)
In [ ]:
bout_data
Out[ ]:
memmap([[[[-1.31044950e+01, -1.07726199e+01, -9.31814325e+00, ...,
           -6.59102380e-01, -6.59102380e-01, -6.59102380e-01],
          [-1.51931409e+01, -1.22127697e+01, -9.59159267e+00, ...,
           -6.62805868e-01, -6.62805868e-01, -6.62805868e-01],
          [-2.27258283e+01, -1.36529194e+01, -9.86504208e+00, ...,
           -6.66509356e-01, -6.66509356e-01, -6.66509356e-01],
          ...,
          [ 1.58692752e+01,  3.32809154e+00,  2.23416430e+00, ...,
            2.44938261e-01,  2.44938261e-01,  2.44938261e-01],
          [ 9.62912943e+00,  2.83513561e+00,  2.23416430e+00, ...,
            2.44938261e-01,  2.44938261e-01,  2.44938261e-01],
          [ 5.19271162e+00,  2.83513561e+00,  2.23416430e+00, ...,
            2.44938261e-01,  2.44938261e-01,  2.44938261e-01]],

         [[-1.51724241e+01, -1.02950914e+01, -8.20474653e+00, ...,
           -6.52507450e-01, -6.52507450e-01, -6.52507450e-01],
          [-1.44053712e+01, -1.14289234e+01, -8.58725953e+00, ...,
           -6.60360707e-01, -6.57826568e-01, -6.56187112e-01],
          [-1.68870379e+01, -1.42346166e+01, -9.05483154e+00, ...,
           -6.64460732e-01, -6.61686425e-01, -6.59866773e-01],
          ...,
          [-3.54836018e+00,  1.46486296e+00,  5.72287187e-01, ...,
            2.37319136e-01,  2.36572991e-01,  2.35982653e-01],
          [-8.85968063e+00,  1.46227195e+00,  6.43827933e-01, ...,
            2.37350424e-01,  2.36506795e-01,  2.35982653e-01],
          [-6.50229845e+00, -1.67490716e+00, -4.54792534e-02, ...,
            2.35982653e-01,  2.35982653e-01,  2.35982653e-01]],

         [[-1.62226139e+01, -9.81756298e+00, -7.09134980e+00, ...,
           -6.45912520e-01, -6.45912520e-01, -6.45912520e-01],
          [-1.38035766e+01, -9.71475911e+00, -7.22248697e+00, ...,
           -6.54930419e-01, -6.51774346e-01, -6.49568355e-01],
          [-1.48065590e+01, -1.24087737e+01, -8.23603428e+00, ...,
           -6.59895328e-01, -6.56056616e-01, -6.53224190e-01],
          ...,
          [-1.36240982e+01, -7.44816370e+00, -3.85904540e+00, ...,
            2.31412517e-01,  2.29372410e-01,  2.27027045e-01],
          [-1.27016182e+01, -8.27758909e+00, -3.86570880e+00, ...,
            2.30730356e-01,  2.28920863e-01,  2.27027045e-01],
          [-9.61784709e+00, -6.18494992e+00, -2.32512281e+00, ...,
            2.27027045e-01,  2.27027045e-01,  2.27027045e-01]],

         ...,

         [[ 2.65698202e+01,  2.61640129e+01,  2.47086401e+01, ...,
            3.48661199e-01,  3.48661199e-01,  3.48661199e-01],
          [ 2.31269413e+01,  2.50436836e+01,  2.44801886e+01, ...,
            3.43999870e-01,  3.45554837e-01,  3.50740827e-01],
          [ 1.77614886e+01,  2.12224817e+01,  2.32910109e+01, ...,
            3.46384715e-01,  3.48258325e-01,  3.52820455e-01],
          ...,
          [-2.73636923e-01, -2.95356138e+00, -7.21513480e+00, ...,
           -5.75435624e-01, -5.74366052e-01, -5.73294373e-01],
          [ 6.68685970e-01, -4.97443986e+00, -8.15128005e+00, ...,
           -5.73772512e-01, -5.73215432e-01, -5.73294373e-01],
          [-1.36614229e+00, -6.29618506e+00, -7.54300828e+00, ...,
           -5.73294373e-01, -5.73294373e-01, -5.73294373e-01]],

         [[ 2.23678771e+01,  2.39879184e+01,  2.47086401e+01, ...,
            3.48661199e-01,  3.48661199e-01,  3.48661199e-01],
          [ 1.80660345e+01,  2.25698427e+01,  2.34568643e+01, ...,
            3.49268518e-01,  3.49212804e-01,  3.50740827e-01],
          [ 1.42964605e+01,  1.92277744e+01,  2.21292769e+01, ...,
            3.51401893e-01,  3.51455124e-01,  3.52820455e-01],
          ...,
          [-3.70921716e+00, -4.50725090e+00, -7.41031154e+00, ...,
           -5.74330316e-01, -5.73708403e-01, -5.73294373e-01],
          [ 5.50775292e+00, -2.39147457e+00, -6.51989304e+00, ...,
           -5.73538680e-01, -5.73158879e-01, -5.73294373e-01],
          [ 2.25973450e+00, -4.69063304e+00, -7.54300828e+00, ...,
           -5.73294373e-01, -5.73294373e-01, -5.73294373e-01]],

         [[ 1.76555365e+01,  2.39879184e+01,  2.47086401e+01, ...,
            3.48661199e-01,  3.48661199e-01,  3.48661199e-01],
          [ 1.69437098e+01,  2.13044874e+01,  2.34641682e+01, ...,
            3.50740827e-01,  3.50740827e-01,  3.50740827e-01],
          [ 1.66954722e+01,  1.86210564e+01,  2.22196962e+01, ...,
            3.52820455e-01,  3.52820455e-01,  3.52820455e-01],
          ...,
          [-4.91528467e+00, -5.82091821e+00, -7.54300828e+00, ...,
           -5.73294373e-01, -5.73294373e-01, -5.73294373e-01],
          [ 4.42433594e+00, -4.69063304e+00, -7.54300828e+00, ...,
           -5.73294373e-01, -5.73294373e-01, -5.73294373e-01],
          [ 2.68535675e+00, -4.69063304e+00, -7.54300828e+00, ...,
           -5.73294373e-01, -5.73294373e-01, -5.73294373e-01]]],


        [[[-2.67670497e+01, -2.25152001e+01, -2.04868811e+01, ...,
           -2.08801030e-01, -2.08801030e-01, -2.08801030e-01],
          [-2.51385478e+01, -2.18916655e+01, -1.83235556e+01, ...,
           -1.97911847e-01, -1.97911847e-01, -1.97911847e-01],
          [-2.85658013e+01, -2.12681308e+01, -1.61602301e+01, ...,
           -1.87022664e-01, -1.87022664e-01, -1.87022664e-01],
          ...,
          [-2.95981882e+01, -1.72691696e+01, -1.21571192e+01, ...,
           -1.02677648e-01, -1.02677648e-01, -1.02677648e-01],
          [-1.71524835e+01, -1.84816307e+01, -1.21571192e+01, ...,
           -1.02677648e-01, -1.02677648e-01, -1.02677648e-01],
          [-2.04185727e+01, -1.84816307e+01, -1.21571192e+01, ...,
           -1.02677648e-01, -1.02677648e-01, -1.02677648e-01]],

         [[-3.11160823e+01, -2.48073797e+01, -2.07034690e+01, ...,
           -2.12516097e-01, -2.12516097e-01, -2.12516097e-01],
          [-2.84270985e+01, -2.40571643e+01, -2.00794420e+01, ...,
           -2.05379378e-01, -2.03295443e-01, -2.01597298e-01],
          [-3.00767080e+01, -2.28881370e+01, -1.93103421e+01, ...,
           -1.95925588e-01, -1.93365675e-01, -1.90678499e-01],
          ...,
          [-2.30762165e+01, -2.04364817e+01, -1.41811789e+01, ...,
           -1.00283153e-01, -1.00380659e-01, -1.03781461e-01],
          [-2.26682784e+01, -2.19042827e+01, -1.37359878e+01, ...,
           -1.03683230e-01, -1.02989762e-01, -1.03781461e-01],
          [-2.94124834e+01, -2.19047512e+01, -1.19250553e+01, ...,
           -1.03781461e-01, -1.03781461e-01, -1.03781461e-01]],

         [[-2.88794802e+01, -2.70995592e+01, -2.09200568e+01, ...,
           -2.16231163e-01, -2.16231163e-01, -2.16231163e-01],
          [-3.01440877e+01, -2.60383186e+01, -2.09649163e+01, ...,
           -2.09334155e-01, -2.06914247e-01, -2.05282749e-01],
          [-3.53707639e+01, -2.63834378e+01, -2.13554601e+01, ...,
           -2.01248779e-01, -1.97478598e-01, -1.94334334e-01],
          ...,
          [-2.79820620e+01, -2.04184069e+01, -1.48595173e+01, ...,
           -1.01777506e-01, -1.01965952e-01, -1.04885274e-01],
          [-4.10431063e+01, -2.49123885e+01, -1.57314997e+01, ...,
           -1.04902272e-01, -1.04234869e-01, -1.04885274e-01],
          [-4.04297586e+01, -2.53278717e+01, -1.16929913e+01, ...,
           -1.04885274e-01, -1.04885274e-01, -1.04885274e-01]],

         ...,

         [[-1.62683325e+01, -1.62658801e+01, -1.48409055e+01, ...,
           -3.18822642e-01, -3.18822642e-01, -3.18822642e-01],
          [-1.62570995e+01, -1.55577437e+01, -1.29127068e+01, ...,
           -3.11377319e-01, -3.09338787e-01, -3.07499417e-01],
          [-2.17673428e+01, -1.77323418e+01, -1.28986179e+01, ...,
           -3.05632632e-01, -3.00944019e-01, -2.96176192e-01],
          ...,
          [-2.76910141e+01, -1.74524268e+01, -1.37194241e+01, ...,
            4.04452301e-02,  3.90627822e-02,  3.52440062e-02],
          [-1.93527245e+01, -1.72753294e+01, -1.27122101e+01, ...,
            3.74224606e-02,  3.67205232e-02,  3.52440062e-02],
          [-5.81451216e+00, -1.52678896e+01, -1.02138230e+01, ...,
            3.52440062e-02,  3.52440062e-02,  3.52440062e-02]],

         [[-2.10050470e+01, -1.77386267e+01, -1.48409055e+01, ...,
           -3.18822642e-01, -3.18822642e-01, -3.18822642e-01],
          [-2.18759322e+01, -1.66845986e+01, -1.36050946e+01, ...,
           -3.10826041e-01, -3.08995063e-01, -3.07499417e-01],
          [-1.87930035e+01, -1.60993191e+01, -1.29446996e+01, ...,
           -3.04191016e-01, -3.00402557e-01, -2.96176192e-01],
          ...,
          [-2.65976314e+01, -1.78504883e+01, -1.33124233e+01, ...,
            3.97576202e-02,  3.94595585e-02,  3.52440062e-02],
          [-1.82518194e+01, -1.52351135e+01, -1.10655539e+01, ...,
            3.72067978e-02,  3.70316129e-02,  3.52440062e-02],
          [-6.83084563e+00, -1.57585155e+01, -1.02138230e+01, ...,
            3.52440062e-02,  3.52440062e-02,  3.52440062e-02]],

         [[-1.91202026e+01, -1.77386267e+01, -1.48409055e+01, ...,
           -3.18822642e-01, -3.18822642e-01, -3.18822642e-01],
          [-2.15633270e+01, -1.72611422e+01, -1.48310588e+01, ...,
           -3.07499417e-01, -3.07499417e-01, -3.07499417e-01],
          [-2.15627099e+01, -1.67836577e+01, -1.48212122e+01, ...,
           -2.96176192e-01, -2.96176192e-01, -2.96176192e-01],
          ...,
          [-1.92399853e+01, -1.32307423e+01, -1.02138230e+01, ...,
            3.52440062e-02,  3.52440062e-02,  3.52440062e-02],
          [-1.76259994e+01, -1.57585155e+01, -1.02138230e+01, ...,
            3.52440062e-02,  3.52440062e-02,  3.52440062e-02],
          [-1.26887218e+01, -1.57585155e+01, -1.02138230e+01, ...,
            3.52440062e-02,  3.52440062e-02,  3.52440062e-02]]],


        [[[-1.25568691e+01, -3.63734765e+00,  4.26934688e-01, ...,
            1.56327637e+00,  1.56327637e+00,  1.56327637e+00],
          [-7.08903640e+00,  5.58251621e-01,  3.97459783e+00, ...,
            1.57197462e+00,  1.57197462e+00,  1.57197462e+00],
          [-1.14758675e+01,  4.75385089e+00,  7.52226097e+00, ...,
            1.58067286e+00,  1.58067286e+00,  1.58067286e+00],
          ...,
          [ 1.47498835e+01, -1.11370350e+00, -2.55596764e+00, ...,
           -1.52850141e+00, -1.52850141e+00, -1.52850141e+00],
          [ 1.78436871e+01,  3.12553805e+00, -2.55596764e+00, ...,
           -1.52850141e+00, -1.52850141e+00, -1.52850141e+00],
          [ 2.31120637e+01,  3.12553805e+00, -2.55596764e+00, ...,
           -1.52850141e+00, -1.52850141e+00, -1.52850141e+00]],

         [[-1.38043562e+01, -3.85059525e+00,  9.03193560e-01, ...,
            1.57321548e+00,  1.57321548e+00,  1.57321548e+00],
          [-3.23470714e+00, -4.16018212e-01,  1.94594457e+00, ...,
            1.58479641e+00,  1.58286120e+00,  1.58201088e+00],
          [-5.65077822e-01,  3.30084924e+00,  5.71149574e+00, ...,
            1.59546865e+00,  1.59253304e+00,  1.59080628e+00],
          ...,
          [-1.17262496e+01, -3.64092337e+00, -4.63374478e+00, ...,
           -1.54373267e+00, -1.53868381e+00, -1.53385555e+00],
          [ 1.50436700e+00,  7.88626084e-01, -1.37803351e+00, ...,
           -1.53964635e+00, -1.53621682e+00, -1.53385555e+00],
          [ 1.58384235e+01,  2.11202417e+00, -1.99169425e+00, ...,
           -1.53385555e+00, -1.53385555e+00, -1.53385555e+00]],

         [[-1.36876646e+01, -4.06384286e+00,  1.37945243e+00, ...,
            1.58315458e+00,  1.58315458e+00,  1.58315458e+00],
          [-6.94444318e+00, -1.34377169e+00,  1.59705613e+00, ...,
            1.59651792e+00,  1.59373550e+00,  1.59204714e+00],
          [-7.15333851e-01,  2.01921434e+00,  3.37710102e+00, ...,
            1.60824144e+00,  1.60404185e+00,  1.60093970e+00],
          ...,
          [-4.20760393e+01, -2.02449911e+01, -1.11003431e+01, ...,
           -1.55366119e+00, -1.54640858e+00, -1.53920968e+00],
          [-1.47993216e+01, -8.91304327e+00, -5.04027288e+00, ...,
           -1.54751576e+00, -1.54280594e+00, -1.53920968e+00],
          [ 1.23213648e+01,  1.09851030e+00, -1.42742086e+00, ...,
           -1.53920968e+00, -1.53920968e+00, -1.53920968e+00]],

         ...,

         [[-9.81503980e+00, -8.90668518e+00, -4.63403331e+00, ...,
            1.81371842e+00,  1.81371842e+00,  1.81371842e+00],
          [-2.57478506e+01, -1.04551020e+01, -3.78519063e+00, ...,
            1.83476476e+00,  1.82946347e+00,  1.82460337e+00],
          [-2.78454464e+01, -1.48304583e+01, -7.98367816e+00, ...,
            1.84959822e+00,  1.84228627e+00,  1.83548832e+00],
          ...,
          [ 3.26097491e+01,  2.20917464e+01,  1.32505260e+01, ...,
           -1.28113336e+00, -1.27370096e+00, -1.26538114e+00],
          [ 4.05441763e+01,  1.96914544e+01,  1.05568383e+01, ...,
           -1.27532480e+00, -1.27040054e+00, -1.26538114e+00],
          [ 2.83989139e+01,  1.61975399e+01,  7.40820167e+00, ...,
           -1.26538114e+00, -1.26538114e+00, -1.26538114e+00]],

         [[-1.76107734e+01, -1.10968855e+01, -4.63403331e+00, ...,
            1.81371842e+00,  1.81371842e+00,  1.81371842e+00],
          [-2.27036649e+01, -1.24252386e+01, -6.11022404e+00, ...,
            1.83050046e+00,  1.82696463e+00,  1.82460337e+00],
          [-2.32000887e+01, -1.48860816e+01, -7.72424970e+00, ...,
            1.84347711e+00,  1.83878906e+00,  1.83548832e+00],
          ...,
          [ 2.31377254e+01,  1.52988707e+01,  9.21385926e+00, ...,
           -1.27513300e+00, -1.27020665e+00, -1.26538114e+00],
          [ 3.99142344e+01,  1.99417678e+01,  1.14976231e+01, ...,
           -1.27148261e+00, -1.26809700e+00, -1.26538114e+00],
          [ 2.90517073e+01,  1.52812838e+01,  7.40820167e+00, ...,
           -1.26538114e+00, -1.26538114e+00, -1.26538114e+00]],

         [[-2.06190927e+01, -1.10968855e+01, -4.63403331e+00, ...,
            1.81371842e+00,  1.81371842e+00,  1.81371842e+00],
          [-2.19604929e+01, -1.11007737e+01, -4.50158876e+00, ...,
            1.82460337e+00,  1.82460337e+00,  1.82460337e+00],
          [-1.82218546e+01, -1.11046619e+01, -4.36914422e+00, ...,
            1.83548832e+00,  1.83548832e+00,  1.83548832e+00],
          ...,
          [ 2.14579532e+01,  1.22652075e+01,  7.40820167e+00, ...,
           -1.26538114e+00, -1.26538114e+00, -1.26538114e+00],
          [ 2.39156764e+01,  1.52812838e+01,  7.40820167e+00, ...,
           -1.26538114e+00, -1.26538114e+00, -1.26538114e+00],
          [ 1.80771953e+01,  1.52812838e+01,  7.40820167e+00, ...,
           -1.26538114e+00, -1.26538114e+00, -1.26538114e+00]]]])

Format Conversion¶

to HDF¶

In [ ]:
r = file.NlfffFile()

h5_path = sample_bout_path.replace(".bin",".h5")

r.tran_bin2hdf5(sample_bout_path, h5_path, nx=sample_nx,ny=sample_ny,nz=sample_nz, overwrite=True)
Out[ ]:
True

Visualization¶

Three-dimensional cutting and drawing¶

Creating drawing objects¶

In [ ]:
from pynlfff.pyplot import plot3d_cut
d3_drawer = plot3d_cut.NlfffPlotD3CutCake()

Set drawing object properties (optional)¶

In [ ]:
# Set the hue range manually 
# Set the maximum display value 
d3_drawer.colormap_set_max=40
# Set the minimum display value 
d3_drawer.colormap_set_max=-30


# Automatically set the hue range 
# The maximum and minimum values in the automatically-sought drawing cut 
d3_drawer.colormap_auto_value=True
# Only effective if colormap_auto_value is True, means compress again on auto_max value, compress ratio, take value range (0,1], 1 means no compress 
d3_drawer.colormap_auto_zip=0.8
# indicates that the value with the largest absolute value is the maximum value, and its opposite is the minimum value, when the palette obtained is symmetric about 0. Note that if the maximum value and the minimum value are both positive or negative, it does not take effect 
d3_drawer.colormap_auto_mirror=True
In [ ]:
# The out-of-bounds value is displayed 
# Default set to True, the out-of-range value is taken according to the closest one; to False, the out-of-range value is not displayed 
d3_drawer.colormap_out_range_display=True
#
In [ ]:
# Transparency 
# Set None to be opaque
d3_drawer.colormap_alpha=None
# Set the overall transparency in the range (0,1)
d3_drawer.colormap_alpha=0.8
# Default, auto auto-transparency, i.e. the smaller the absolute value the higher the degree of transparency 
d3_drawer.colormap_alpha="auto"
In [ ]:
# Set the tangent boundary 
# Do not draw boundary lines 
d3_drawer.cut_line_edges = None
# Default, set boundary line style 
d3_drawer.cut_line_edges = dict(color='0.6', linewidth=0.6, zorder=1e3)
In [ ]:
# Set tangent intersection line 
# Don't draw cut_line_cross 
d3_drawer.cut_line_cross = None
# Default, set cut line cross style 
d3_drawer.cut_line_cross = dict(color='0.4', linewidth=1, zorder=1e3)

Load Data¶

In [ ]:
# Configuration Data
use_facker_data=False
if use_facker_data:
    import numpy as np
    array_data = np.random.uniform(-5, 5, size=(3, 100,50,40))
else:
    array_data=bout_data.copy()
In [ ]:
# Loading data
d3_drawer.load_data_array(array_data)
Out[ ]:
True

Adding a drawing cut¶

In [ ]:
d3_drawer.add_cut(B="Bx", N="Nx", cut_num=None, cut_percent=0.5)
d3_drawer.add_cut("Bz", "Nz")
d3_drawer.add_cut("Bz", "Nz", cut_num=0)
d3_drawer.add_cut("Bz", "Nx",cut_percent=0.3)
d3_drawer.add_cut("Bz", "Nz", cut_percent=0.2)
# add cut surface for cut draw, which can be use many times for different surfance in one picture
# :param B: 'Bx' 'By' or 'Bz'; choose one Magnetic field component to draw
# :param N: 'Nx' 'Ny' or 'Nz'; choose cut in which angle; when choose 'Nz', The slice will be perpendicular to z axis
# :param cut_num: int type , set for cut where, can be [0, max(axis)], if not set this,will use cut_percent default
# :param cut_percent: default 0.5, percent of axis

Performing drawing operations¶

In [ ]:
# Configuration Information

# Canvas Settings https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.figure.html
fig_dict=dict(figsize=[6, 6])

# Save image settings https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.savefig.html
is_save=False
save_dict=dict(fname="./cut.png", dpi='figure')

# Display settings https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.show.html
is_show=True
show_dict=dict(block=True)

# view setting
view_dict=dict(
    elev=45,
    azim=45,
    dist=11
)

d3_drawer.run_cut(
    fig_dict=fig_dict,
    is_show=is_show,
    view_dict=view_dict,
    show_dict=show_dict,
    is_save=is_save,
    save_dict=save_dict
)

Three-dimensional body drawing¶

In [ ]:
 # Save directory 
out_dir=". /" 

# Does the system come with gui
is_with_gui=True # System with gui
is_with_gui=False # System without gui like colab


# Configuration data 
use_facker_data=True
if use_facker_data:
    import numpy as np
    array_data = np.random.uniform(-5, 5, size=(3, 100,50,40))
else:
    array_data=bout_data.copy()
In [ ]:
if is_with_gui:
  from pynlfff.pyplot import plot3d_body
  import numpy as np
  
  try:
    dataBout=array_data
    result=plot3d_body.main_draw_3d_body(
        dataBout,
        out_dir,
        axis_name=['bx','by','bz']
    )
  except BaseException as e:
    print(e)
else:
  print("VTK Drawing dependency gui")

VTK Drawing dependency gui

More sample acquisition methods¶

Method 1 Download by checking the official website¶

Visit the download page of https://nlfff.dataset.deepsolar.space/ and find the online query download related website

Method 2 Jump to view samples online via identifiers¶

In [ ]:
# sample identifiers can be obtained from inside the product information table 

sample_identifiers=7327201812200936003
# where 7327(harp number)20181220(date)093600(time)3(Max grid level). 
sample_url="https://identifiers.org/nlfff:{}".format(sample_identifiers)
print("Visit {} for sample details and downloads".format(sample_url))

Visit https://identifiers.org/nlfff:7327201812200936003 for sample details and downloads

The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click <a href='https://aka.ms/vscodeJupyterKernelCrash'>here</a> for more info. View Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details.

Method 3 Online request for hard copy by mail¶

For researchers who need to use a large number of samples, we recommend using the hard disk mailing method.

You can visit visit the contact page at https://nlfff.dataset.deepsolar.space/ to communicate specific details such as mailing address recipient via email.