How the algorithms such as Xgboost and LightGBM fills the NAN ? I know that they learn how to fill it but how they do it is my question and what is they method of learning how to fill the NAN?
LIGHTGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. Section 3.2 of this reference explains it.
There are some options you can set such as usemissing=false, which disables handling for missing values. You can also use the zeroas_missing option to change behavior. GitHub