大師兄的Python源碼學(xué)習(xí)筆記(三十六）: 模塊的動(dòng)態(tài)加載機(jī)制（三）
大師兄的Python源碼學(xué)習(xí)筆記(三十八）: 模塊的動(dòng)態(tài)加載機(jī)制（五）
三弧岳、import機(jī)制的實(shí)現(xiàn)
1. 找到module/package
- 這一部分實(shí)際分析了大師兄的Python源碼學(xué)習(xí)筆記(三十四）: 模塊的動(dòng)態(tài)加載機(jī)制（一）中蹋盆，PyImport_ImportModuleLevelObject函數(shù)到PyImport_GetModule函數(shù)之間發(fā)生的故事因妙。
- 回到熟悉的PyImport_ImportModuleLevelObject函數(shù)：
Python\import.c
PyObject *
PyImport_ImportModuleLevelObject(PyObject *name, PyObject *globals,
PyObject *locals, PyObject *fromlist,
int level)
{
_Py_IDENTIFIER(_handle_fromlist);
PyObject *abs_name = NULL;
PyObject *final_mod = NULL;
PyObject *mod = NULL;
PyObject *package = NULL;
PyInterpreterState *interp = PyThreadState_GET()->interp;
int has_from;
... ...
mod = import_find_and_load(abs_name);
... ...
- 在經(jīng)過(guò)一系列判斷后蚂斤，調(diào)用核心函數(shù)import_find_and_load：
Python\import.c
static PyObject *
import_find_and_load(PyObject *abs_name)
{
_Py_IDENTIFIER(_find_and_load);
PyObject *mod = NULL;
PyInterpreterState *interp = PyThreadState_GET()->interp;
int import_time = interp->core_config.import_time;
static int import_level;
static _PyTime_t accumulated;
... ...
mod = _PyObject_CallMethodIdObjArgs(interp->importlib,
&PyId__find_and_load, abs_name,
interp->import_func, NULL);
... ...
return mod;
}
- 跳轉(zhuǎn)到Python并調(diào)用Python函數(shù)_find_and_load_unlocked:
Lib\importlib\_bootstrap.py
def _find_and_load_unlocked(name, import_):
path = None
parent = name.rpartition('.')[0]
if parent:
if parent not in sys.modules:
_call_with_frames_removed(import_, parent)
# Crazy side-effects!
if name in sys.modules:
return sys.modules[name]
parent_module = sys.modules[parent]
try:
path = parent_module.__path__
except AttributeError:
msg = (_ERR_MSG + '; {!r} is not a package').format(name, parent)
raise ModuleNotFoundError(msg, name=name) from None
spec = _find_spec(name, path)
if spec is None:
raise ModuleNotFoundError(_ERR_MSG.format(name), name=name)
else:
module = _load_unlocked(spec)
if parent:
# Set the module as an attribute on its parent.
parent_module = sys.modules[parent]
setattr(parent_module, name.rpartition('.')[2], module)
return module
- _find_and_load_unlocked函數(shù)會(huì)通過(guò)_find_spec函數(shù)找到需要加載的模塊在哪里，然后調(diào)用_load_unlocked函數(shù)來(lái)加載模塊庙洼。
- 首先仔細(xì)觀察_find_spec函數(shù)：
Lib\importlib\_bootstrap.py
def _find_spec(name, path, target=None):
"""Find a module's spec."""
meta_path = sys.meta_path
if meta_path is None:
# PyImport_Cleanup() is running or has been called.
raise ImportError("sys.meta_path is None, Python is likely "
"shutting down")
if not meta_path:
_warnings.warn('sys.meta_path is empty', ImportWarning)
# We check sys.modules here for the reload case. While a passed-in
# target will usually indicate a reload there is no guarantee, whereas
# sys.modules provides one.
is_reload = name in sys.modules
for finder in meta_path:
with _ImportLockContext():
try:
find_spec = finder.find_spec
except AttributeError:
spec = _find_spec_legacy(finder, name, path)
if spec is None:
continue
else:
spec = find_spec(name, path, target)
if spec is not None:
# The parent import may have already imported this module.
if not is_reload and name in sys.modules:
module = sys.modules[name]
try:
__spec__ = module.__spec__
except AttributeError:
# We use the found spec since that is the one that
# we would have used if the parent module hadn't
# beaten us to the punch.
return spec
else:
if __spec__ is None:
return spec
else:
return __spec__
else:
return spec
else:
return None
- _find_spec函數(shù)會(huì)遍歷sys.meta_path, 對(duì)其中的每一個(gè)對(duì)象調(diào)用findspec函數(shù)培己。
- 查看sys.meta_path：
>>> sys.meta_path
[<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>]
- Python模塊按導(dǎo)入方式一共分為六種：
1.builtin module loader
2.frozen module loader
3.source file loader
4.sourceless file loader
5.extension module loader
6.zipimport loader
- 可以看到sys.meta_path包含的三個(gè)對(duì)象: 其中BuiltinImporter和FrozenImporter分別對(duì)應(yīng)模塊加載的前兩種方法依溯，而后四種加載方法都是由PathFinder實(shí)現(xiàn)的。
- 查看PathFinder中的find_spec方法：
Lib\importlib\_bootstrap_external.py
class PathFinder:
"""Meta path finder for sys.path and package __path__ attributes."""
... ...
@classmethod
def find_spec(cls, fullname, path=None, target=None):
"""Try to find a spec for 'fullname' on sys.path or 'path'.
The search is based on sys.path_hooks and sys.path_importer_cache.
"""
if path is None:
path = sys.path
spec = cls._get_spec(fullname, path, target)
if spec is None:
return None
elif spec.loader is None:
namespace_path = spec.submodule_search_locations
if namespace_path:
# We found at least one namespace path. Return a spec which
# can create the namespace package.
spec.origin = None
spec.submodule_search_locations = _NamespacePath(fullname, namespace_path, cls._get_spec)
return spec
else:
return None
else:
return spec
... ...
- PathFinder.findspec會(huì)對(duì)sys.path_hooks里的每一個(gè)hook函數(shù)分別調(diào)用sys.path里的每一個(gè)路徑葡粒。
- 如果某一個(gè)hook函數(shù)在指定的路徑下發(fā)現(xiàn)了指定的包份殿，它就會(huì)返回一個(gè)_bootstrap.ModuleSpec對(duì)象膜钓。
- 在_bootstrap.ModuleSpec中會(huì)有一個(gè)loader對(duì)象負(fù)責(zé)真正加載這個(gè)包：
Lib\importlib\_bootstrap_external.py
def _get_supported_file_loaders():
"""Returns a list of file-based module loaders.
Each item is a tuple (loader, suffixes).
"""
extensions = ExtensionFileLoader, _imp.extension_suffixes()
source = SourceFileLoader, SOURCE_SUFFIXES
bytecode = SourcelessFileLoader, BYTECODE_SUFFIXES
return [extensions, source, bytecode]
- PathFinder返回的ModuleSpec可能的loader有以下三種:ExtensionFileLoader, SourceFileLoader, SourcelessFileLoader。
- 接著觀察_load_unlocked函數(shù)：
Lib\importlib\_bootstrap.py
def _load_unlocked(spec):
# A helper for direct use by the import system.
if spec.loader is not None:
# not a namespace package
if not hasattr(spec.loader, 'exec_module'):
return _load_backward_compatible(spec)
module = module_from_spec(spec)
with _installed_safely(module):
if spec.loader is None:
if spec.submodule_search_locations is None:
raise ImportError('missing loader', name=spec.name)
# A namespace package so do nothing.
else:
spec.loader.exec_module(module)
# We don't ensure that the import-related module attributes get
# set in the sys.modules replacement case. Such modules are on
# their own.
return sys.modules[spec.name]
- _load_unlocked函數(shù)創(chuàng)建了一個(gè)空的module對(duì)象卿嘲，然后將這個(gè)空moudle對(duì)象作為參數(shù)調(diào)用了具體loader的exec_module函數(shù)颂斜。
- 回到作為exec_module函數(shù)的ExtensionFileLoader類：
Lib\importlib\_bootstrap_external.py
class ExtensionFileLoader(FileLoader, _LoaderBasics):
"""Loader for extension modules.
The constructor is designed to work with FileFinder.
"""
... ...
def create_module(self, spec):
"""Create an unitialized extension module"""
module = _bootstrap._call_with_frames_removed(
_imp.create_dynamic, spec)
_bootstrap._verbose_message('extension module {!r} loaded from {!r}',
spec.name, self.path)
return module
... ...
- 在ExtensionFileLoader中調(diào)用了builtin對(duì)象_imp.create_dynamic,它的實(shí)現(xiàn)在C語(yǔ)言函數(shù)_imp_create_dynamic_impl：
Python\import.c
static PyObject *
_imp_create_dynamic_impl(PyObject *module, PyObject *spec, PyObject *file)
/*[clinic end generated code: output=83249b827a4fde77 input=c31b954f4cf4e09d]*/
{
PyObject *mod, *name, *path;
FILE *fp;
... ...
mod = _PyImport_LoadDynamicModuleWithSpec(spec, fp);
Py_DECREF(name);
Py_DECREF(path);
if (fp)
fclose(fp);
return mod;
}
- 最后進(jìn)入_PyImport_LoadDynamicModuleWithSpec函數(shù)：
PyObject *
_PyImport_LoadDynamicModuleWithSpec(PyObject *spec, FILE *fp)
{
... ...
#ifdef MS_WINDOWS
exportfunc = _PyImport_FindSharedFuncptrWindows(hook_prefix, name_buf,
path, fp);
#else
pathbytes = PyUnicode_EncodeFSDefault(path);
if (pathbytes == NULL)
goto error;
exportfunc = _PyImport_FindSharedFuncptr(hook_prefix, name_buf,
PyBytes_AS_STRING(pathbytes),
fp);
Py_DECREF(pathbytes);
#endif
... ...
p0 = (PyObject *(*)(void))exportfunc;
/* Package context is needed for single-phase init */
oldcontext = _Py_PackageContext;
_Py_PackageContext = PyUnicode_AsUTF8(name_unicode);
if (_Py_PackageContext == NULL) {
_Py_PackageContext = oldcontext;
goto error;
}
m = p0();
... ...
}
- 最終程序?qū)⒄{(diào)用dlopen/LoadLibrary來(lái)加載動(dòng)態(tài)鏈接庫(kù)并且執(zhí)行其中的PyInit_modulename。
