前些天在看David Beazley的Python3 metaprogramming 视频,觉得是时候总结下视频中学到的内容, 这篇文章主要是这个视频的笔记,以及关于metaclass的一些思考.
装饰器 首先看一个需求,就是想在函数被调用时,记录一下,可以简单的加个print
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 def  add (x, y) :    print('add' )     return  x + y def  sub (x, y) :    print('sub' )     return  x - y def  mul (x, y) :    print('mul' )     return  x * y def  div (x, y) :    print('div' )     return  x / y print(add(3 , 5 )) print(sub(5 , 2 )) 输出 add 8 sub 3 
但是这样的话, print语句就重复了, 每个函数里都得加print, 于是我们可以使用装饰器
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 def  debug (func) :    def  wrapper (*args, **kwargs) :         print(func.__name__)         return  func(*args, **kwargs)     return  wrapper @debug def  add (x, y) :    return  x + y @debug def  sub (x, y) :    return  x - y @debug def  mul (x, y) :    return  x * y @debug def  div (x, y) :    return  x / y print(add(3 , 5 )) print(sub(5 , 2 )) 输出 add 8 sub 3 
但是这个简单的装饰器是存在问题的,它会忽略被装饰的函数
1 2 3 4 print (add) 输出 <function debug.<locals>.wrapper  at 0 x1037a6bf8> 
这里add函数经过debug装饰器装饰后,函数名都被忽略了,这个时候functools模块就派上用场了, 里面的wraps装饰器就是用来解决这个问题
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 from  functools import  wrapsdef  debug (func) :    msg = func.__qualname__     @wraps(func)     def  wrapper (*args, **kwargs) :         print(msg)         return  func(*args, **kwargs)     return  wrapper      @debug def  add (x, y) :    return  x + y @debug def  sub (x, y) :    return  x - y @debug def  mul (x, y) :    return  x * y @debug def  div (x, y) :    return  x / y print(add(3 , 5 )) print(sub(5 , 2 )) 输出 add 8 sub 3 print(add) 输出 <function add at 0x1037afa60 > 
带参数的装饰器 有时候装饰器里想传入一些参数, 这时就可以写带参数的装饰器1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 from  functools import  wrapsdef  debug (prefix='' ) :    def  decroate (func) :         msg = prefix + func.__qualname__         @wraps(func)         def  wrapper (*args, **kwargs) :             print(msg)             return  func(*args, **kwargs)         return  wrapper     return  decroate @debug('###') def  add (x, y) :    return  x + y print(add(3 , 2 )) 输出 5 
这种带参数的装饰器有一个头疼的问题是, 使用装饰器时, 如果不想传参数, 也得加上括号, 不然会报错
1 2 3 4 5 6 7 8 9 10 11 12 13 14 @debug def sub (x, y)      return x - y sub (5 , 3 ) --------------------------------------------------------------------------- TypeError                                 Traceback (most recent call last) <ipython-input-40 -9 a93be17056c> in  <module>       3      return x - y       4   ----> 5  sub (5 , 3 )  TypeError: decroate () 1  positional argument but 2  were given 
加上括号后, 就不报错, 这很丑陋1 2 3 4 5 @debug() def  sub (x, y) :    return  x - y print(sub(5 , 3 )) 
这里有一个小技巧, 实现如下1 2 3 4 5 6 7 8 9 10 11 from  functools import  wraps, partialdef  debug (func=None, prefix='' ) :    if  func is  None :         return  partial(debug, prefix=prefix)     msg = prefix + func.__qualname__     @wraps(func)     def  wrapper (*args, **kwargs) :         print(msg)         return  func(*args, **kwargs)     return  wrapper 
如此, 当使用默认参数时, 即便不带括号时,也不会报错1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 @debug(prefix='###') def  add (x, y) :    return  x + y @debug def  sub (x, y) :    return  x - y print(add(3 , 2 )) print(sub(5 , 3 )) 输出 5 sub 2 
类装饰器 以下我们定义一个Spam类,1 2 3 4 5 6 7 8 9 10 11 class  Spam :    def  a (self) :         pass      def  b (self) :         pass      @classmethod     def  c (cls) :         pass      @staticmethod     def  d () :         pass  
然后我们想在类的方法被调用时, 能够记录下, 想上面的函数被调用时一样, 这时我们就会可以编写一个类装饰器1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 def  debugmethods (cls) :         for  key, val in  vars(cls).items():         if  callable(val):             setattr(cls, key, debug(val))     return  cls @debugmethods class  Spam :    def  a (self) :         pass      def  b (self) :         pass      @classmethod     def  c (cls) :         pass      @staticmethod     def  d () :         pass  spam = Spam() spam.a() spam.b() spam.c() spam.d() 输出 Spam.a Spam.b 
这里只打印了a和b, 没有打印c和d, 这什么原因呢? 这是因为classmethod和staticmethod都是descriptor, 也就是描述器, 它们没有实现__call__方法,也就不是callable的
我们也可以编写一个类装饰器, 当获取一个属性时, 打印日志1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 def  debugattr (cls) :    orig_getattribute = cls.__getattribute__     def  __getattribute__ (self, name) :         print('Get:' , name)         return  orig_getattribute(self, name)     cls.__getattribute__ = __getattribute__     return  cls @debugattr class  Spam :    def  __init__ (self, x, y) :         self.x = x         self.y = y spam = Spam(2 , 3 ) print(spam.x) 输出 Get: x 2 
现在我们想对所有的类都能打印日志,一个解决的办法是在所有的类前面都加上类装饰器1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 @debugmethods class  Base :    def  a (self) :         pass      def  b (self) :         pass       @debugmethods class  Spam (Base) :    def  a (self) :         pass  b = Base() b.a() s = Spam() s.a() 输出 Base.a Spam.a 
但这样很麻烦,于是metaclass派上用场了, metaclass最强的地方是可以控制类的创建
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 class  debugmeta (type) :    def  __new__ (cls, clsname, bases, clsdict) :         clsobj = super().__new__(cls, clsname, bases, clsdict)         clsobj = debugmethods(clsobj)         return  clsobj class  Base (metaclass=debugmeta) :    def  a (self) :         pass      def  b (self) :         pass       class  Spam (Base) :    def  __init__ (self, name) :         self.name = name              def  a (self) :         pass       b = Base() b.a() s = Spam('name' ) s.a() 输出 Base.a Spam.__init__ Spam.a 
从上面的例子中我们看到,有一个类有metaclass, 它的所有子类都有metaclass, 这说明metaclass是会被继承的。结合蔡元楠的《metaclass, 是潘多拉魔盒还是阿拉丁神灯》, 可以知道,这里debugmeta其实不只一种写法,在__init__函数里实现也是可以的。重载__init__的实现如下1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 class  debugmeta (type) :     def  __init__ (cls, name, bases, kwds) :         super(debugmeta, cls).__init__(name, bases, kwds)         cls = debugmethods(cls) class  Base (metaclass=debugmeta) :    def  a (self) :         pass      def  b (self) :         pass       class  Spam (Base) :    def  __init__ (self, name) :         self.name = name              def  a (self) :         pass  b = Base() b.a() s = Spam('lala' ) s.a() 输出 Base.a Spam.__init__ Spam.a 
这是因为, 所有的类都是type的实例, 都是对type的__call__方法进行重载, 而type的__call__方法会调用type.__new__(typeclass, classname, superclasses, attributedict)和type.__init__(class, classname, superclasses, attributedict), 所以上面重写__new__和重写__init__都是可以的。
学到这里,我脑海里冒出了一个想法,就是为啥这里一定要用metaclass呢? 用继承的方式难道不行吗?于是自己尝试写了个继承的方式, 发现也是跑得通的。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 class  debugmeta :    def  __new__ (cls, *args, **kwargs) :         cls = debugmethods(cls)         clsobj = object.__new__(cls)         return  clsobj      class  Base (debugmeta) :    def  a (self) :         pass       class  Spam (Base) :    def  __init__ (self, name) :         self.name = name     def  a (self) :         pass       b = Base() b.a() s = Spam('name' ) s.a() 输出 Base.a Spam.__init__ Spam.a 
但实际上,这样做法是有问题的,后面等到后面我们来纠正这个问题。
既然如此,那么蔡元楠在《metaclass, 是潘多拉魔盒还是阿拉丁神灯》介绍的yaml的动态序列化和逆序列化的能力又为何要用metaclass实现呢?用继承难道不行吗?于是也写了一个继承的版本, 代码如下。1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 import  yamlclass  MyYAMLObjectBaseclass (object) :    """     The metaclass for YAMLObject.     """      def  __new__ (cls, *args, **kwargs) :         if  cls.yaml_tag:             cls.yaml_loader.add_constructor(cls.yaml_tag, cls.from_yaml)             cls.yaml_dumper.add_representer(cls, cls.to_yaml)         return  object.__new__(cls) class  MyYAMLObject (MyYAMLObjectBaseclass) :    """     An object that can dump itself to a YAML stream     and load itself from a YAML stream.     """      __slots__ = ()       yaml_loader = yaml.Loader     yaml_dumper = yaml.Dumper     yaml_tag = None      yaml_flow_style = None      @classmethod     def  from_yaml (cls, loader, node) :         """         Convert a representation node to a Python object.         """          return  loader.construct_yaml_object(node, cls)     @classmethod     def  to_yaml (cls, dumper, data) :         """         Convert a Python object to a representation node.         """          return  dumper.represent_yaml_object(cls.yaml_tag, data, cls,                 flow_style=cls.yaml_flow_style) class  Monster (MyYAMLObject) :    yaml_tag = '!Monster'      def  __init__ (self, name, hp, ac, attacks) :         self.name = name         self.hp = hp         self.ac = ac         self.attacks = attacks     def  __repr__ (self) :         return  "{}(name={}, hp={}, ac={}, attacks={}" .format(self.__class__.__name__, self.name, self.hp,                                                              self.ac, self.attacks) class  Dragon (Monster) :    yaml_tag = '!Dragon'      def  __init__ (self, name, hp, ac, attacks, energy) :         super(Dragon, self).__init__(name, hp, ac, attacks)         self.energy = energy     def  __repr__ (self) :         return  "{}(name={}, hp={}, ac={}, attacks={}, energy={})" .format(self.__class__.__name__, self.name, self.hp,                                                              self.ac, self.attacks, self.energy) m = Monster(name='Cave spider' , hp=[2 , 6 ], ac=16 , attacks=['BITE' , 'HURT' ]) ms = yaml.dump(m) d = Dragon(name='Cave spider' , hp=[2 , 6 ], ac=17 , attacks=['BITE' , 'HURT' ], energy=5000 ) ds = yaml.dump(d) print(yaml.load(ms, Loader=yaml.Loader)) print(yaml.load(ds, Loader=yaml.Loader)) 输出 Monster(name=Cave spider, hp=[2 , 6 ], ac=16 , attacks=['BITE' , 'HURT' ] Dragon(name=Cave spider, hp=[2 , 6 ], ac=17 , attacks=['BITE' , 'HURT' ], energy=5000 ) 
所以其实到这里我还是没有明白为啥yaml要用metaclass来实现这个功能, 直到把子类的yaml_tag去掉, 才发现问题。1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 import yaml class  Monster (MyYAMLObject ):    yaml_tag = '!Monster'      def  __init__ (self , name, hp, ac, attacks) :         self .name = name         self .hp = hp         self .ac = ac         self .attacks = attacks     def  __repr__ (self ) :         return  "{}(name={}, hp={}, ac={}, attacks={}" .format(self .__class__ .__name__ , self .name, self .hp,                                                              self .ac, self .attacks) class  Dragon (Monster ):    def  __init__ (self , name, hp, ac, attacks, energy) :         super (Dragon , self ).__init__ (name, hp, ac, attacks)         self .energy = energy     def  __repr__ (self ) :         return  "{}(name={}, hp={}, ac={}, attacks={}, energy={})" .format(self .__class__ .__name__ , self .name, self .hp,                                                              self .ac, self .attacks, self .energy) m = Monster (name='Cave spider' , hp=[2 , 6 ], ac=16 , attacks=['BITE' , 'HURT' ]) ms = yaml.dump(m) d = Dragon (name='Cave spider' , hp=[2 , 6 ], ac=17 , attacks=['BITE' , 'HURT' ], energy=5000 ) ds = yaml.dump(d) print(yaml.load(ms, Loader =yaml.Loader )) print(yaml.load(ds, Loader =yaml.Loader )) 输出 --------------------------------------------------------------------------- AttributeError                             Traceback  (most recent call last)<ipython-input-58 -1 e3c4c44fbaf> in  <module >      33        34   ---> 35  print(yaml.load(ms, Loader =yaml.Loader ))      36  print(yaml.load(ds, Loader =yaml.Loader )) <ipython-input-58 -1 e3c4c44fbaf> in  __repr__ (self )      24      def  __repr__ (self ) :      25          return  "{}(name={}, hp={}, ac={}, attacks={}, energy={})" .format(self .__class__ .__name__ , self .name, self .hp, ---> 26                                                               self .ac, self .attacks, self .energy)      27        28  m = Monster (name='Cave spider' , hp=[2 , 6 ], ac=16 , attacks=['BITE' , 'HURT' ]) AttributeError :  'Dragon'  object has no attribute 'energy' 
这是因为Dragon这里没有yaml_tag的时候, 会继承父类Monster的yaml_tag, 这时Dragon.yaml_tag就是非空, 然后就会将!Monster这个标记与Dragon绑定到一起了, 覆盖了前面!Monster与Monster的绑定, 这时再去加载Monster类dump出来的内容, 就会报没有energy. 而使用metaclass就不存在这个问题, 因为在创建Dragon类时, 传入的属性字典里不会带有yaml_tag, 也就不会将!Monster这个标记与Dragon绑定到一起. 写代码测试如下1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 import  yamlclass  MyYAMLObjectMetaclass (type ):    """     The metaclass for YAMLObject.     """      def  __new__ (         print(cls, name, bases, kwds.get('yaml_ta g', 'hahah a'))         clsobj = super ().__new__(cls, name, bases, kwds)         if  'yaml_ta g' in kwds and kwds['yaml_ta g'] is not None :             clsobj.yaml_loader.add_constructor(clsobj.yaml_tag, clsobj.from_yaml)             clsobj.yaml_dumper.add_representer(clsobj, clsobj.to_yaml)         return  clsobj class  ThisYAMLObjectMetaclass (type ):    """     The metaclass for YAMLObject.     """      def  __init__ (         print(cls, name, bases, kwds.get('yaml_ta g', 'hahah a'))         super (ThisYAMLObjectMetaclass , cls).__init__(name, bases, kwds)         if  'yaml_ta g' in kwds and kwds['yaml_ta g'] is not None :             cls.yaml_loader.add_constructor(cls.yaml_tag, cls.from_yaml)             cls.yaml_dumper.add_representer(cls, cls.to_yaml) class  ThisYAMLObjectBaseclass (object ):    """     The metaclass for YAMLObject.     """      def  __new__ (         print('base  __new__', cls, args, kwargs)         if  cls.yaml_tag:             cls.yaml_loader.add_constructor(cls.yaml_tag, cls.from_yaml)             cls.yaml_dumper.add_representer(cls, cls.to_yaml)         return  object .__new__ ( class  MyYAMLObject (ThisYAMLObjectMetaclass ):# class  MyYAMLObject (ThisYAMLObjectBaseclass ): # class  MyYAMLObject (MyYAMLObjectMetaclass ):     """     An object that can dump itself to a YAML stream     and load itself from a YAML stream.     """      __slots__ = ()  # no direct instantiation, so allow immutable subclasses     yaml_loader = yaml.Loader      yaml_dumper = yaml.Dumper      yaml_tag = None      yaml_flow_style = None      @classmethod      def  from_yaml (         """         Convert a representation node to a Python object.         """          return  loader.construct_yaml_object(node, cls)     @classmethod      def  to_yaml (         """         Convert a Python object to a representation node.         """          return  dumper.represent_yaml_object(cls.yaml_tag, data, cls,                 flow_style=cls.yaml_flow_style) class  Monster (MyYAMLObject ):    yaml_tag = '!Monster '     def  __init__ (         self.name = name         self.hp = hp         self.ac = ac         self.attacks = attacks     def  __repr__ (         return  "{}(name={}, hp={}, ac={}, attacks={}" .format(self.__class__.__name__, self.name, self.hp,                                                              self.ac, self.attacks) class  Dragon (Monster ):    # yaml_tag = '!Dragon '     def  __init__ (         super (Dragon , self).__init__(name, hp, ac, attacks)         self.energy = energy     def  __repr__ (         return  "{}(name={}, hp={}, ac={}, attacks={}, energy={})" .format(self.__class__.__name__, self.name, self.hp,                                                              self.ac, self.attacks, self.energy) m = Monster (name='Cave  spider', hp=[2 , 6 ], ac=16 , attacks=['BIT E', 'HUR T']) ms = yaml.dump(m) # print(ms) d = Dragon (name='Cave  spider', hp=[2 , 6 ], ac=17 , attacks=['BIT E', 'HUR T'], energy=5000 ) ds = yaml.dump(d) # print(ds) print(yaml.load(ms, Loader =yaml.Loader )) print(yaml.load(ds, Loader =yaml.Loader )) 输出 <class  '__main__ .MyYAMLObject '>  MyYAMLObject  ('__module_ _': '__main_ _', '__qualname_ _': 'MyYAMLObjec t', '__doc_ _': '\n    An  object  that  can  dump  itself  to  a  YAML  stream\n     and  load  itself  from  a  YAML  stream .\n     ', '__slots__ ':'yaml_loade r': <class  'yaml .loader .Loader '> , 'yaml_dumper ':class  'yaml .dumper .Dumper '> , 'yaml_tag ':None , 'yaml_flow_styl e': None , 'from_yam l': <classmethod object  at  0x1038e9320> , 'to_yaml ':object  at  0x1038e9390> }class  '__main__ .Monster '>  Monster  (class  '__main__ .MyYAMLObject '> ,) {'__module_ _': '__main_ _', '__qualname_ _': 'Monste r', 'yaml_ta g': '!Monster ', '__init_ _': <function Monster .__init__ at 0x1038d2f28 >, '__repr_ _': <function Monster .__repr__ at 0x1038ce048 >} <class  '__main__ .Dragon '>  Dragon  (class  '__main__ .Monster '> ,) {'__module_ _': '__main_ _', '__qualname_ _': 'Drago n', '__init_ _': <function Dragon .__init__ at 0x1038ce0d0 >, '__repr_ _': <function Dragon .__repr__ at 0x1038ce158 >, '__classcell_ _': <cell at 0x103837af8 : ThisYAMLObjectMetaclass  object  at  0x7fc8fb130de8> }Monster (name=Cave  spider, hp=[2 , 6 ], ac=16 , attacks=['BIT E', 'HUR T'] Dragon (name=Cave  spider, hp=[2 , 6 ], ac=17 , attacks=['BIT E', 'HUR T'], energy=5000 )
从上面的结果里可以看到, yaml_tag是没有在Dragon类的属性字典里的,即便是Dragon类会从Monster那里继承yaml_tag.
回到前面的用基类来实现对所有类使用debugmethods进行装饰的例子。这里因为每次创建对象的时候都会调用__new__方法, 会导致多次调用debugmethods装饰器, 这样会导致创建多少个对象, 调用一次类的方法就会输出多次, 测试如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 class  debugmeta :    def  __new__ (cls, *args, **kwargs) :         cls = debugmethods(cls)         clsobj = object.__new__ (cls)         return  clsobj      class  Base (debugmeta ):    def  a (self ) :         pass      class  Spam (Base ):    def  __init__ (self , name) :         self .name = name     def  a (self ) :         pass      b = Base () b.a() s = Spam ('name' ) s.a() print('-------' ) s = Spam ('lblb' ) s.a() 输出如下 here <class  '__main__ .Base '> Base .ahere <class  '__main__ .Spam '> Spam .__init__ Spam .a------- here <class  '__main__ .Spam '> Spam .__init__ Spam .__init__ Spam .aSpam .a
这里Spam类创建了两个对象, 就调用了两次debugmethods, 所以会有多次输出。
到了这里, 我终于明白为什么要用metaclass来解决对于所有类使用debugmethods来装饰的问题,因为在metaclass里实现,则只会调用一次,因为一个类的创建只需要一次。也明白为什么yaml要使用metaclass, 而不是继承了。
总的来说, metaclass并不是什么奇淫巧技,简单来说就是一种改变类创建过程的能力。当然, 绝大多数情况下都不需要用到它。